Arduino is pretty handy for low powered projects and is cost-effective (against Raspberry Pi). Not being able to connect to a network natively is a big challenge in applying Arduino for most use cases. While, Arduino has a Ethernet Shield for quite a while, it is impractical to expect a Ethernet port at every point an Arduino would be required. I do have Ethernet-over-Power (EoP) at home to extend its reach, but still didn't scale for me (the over all cost will also include a EoP adapter).
ESP8266 is an effective way to connect Arduino to a Wifi network. It is not a Arduino specific shield. It is a general purpose module that could be mixed with any other micro controller unit/circuit. All you need is a UART communication support on the other end (which almost all micro controllers have built-in). The added advantage is the cost, form-factor (unlike other shields) and just uses 2 pins from the Arduino (RX/TX of UART). The ESP8266 module is pretty small, but does its job well.
Once powered appropriately, you could communicate with the ESP8266 module via serial communication (@9600 baudrate, at least my version of ESP8266). The Wifi configuration/data everything is then communicated via the extensions to a standard AT command set. This being UART based, its going to be imposing restrictions on the rate of transfer of data (but might not matter for most use-cases) -- just to highlight the difference against a USB or SoC based Wifi implementation.
The module does have a built-in TCP/IP stack, so we don't have to worry about that -- that's why we do get response to ping automatically (as shown in the video below). It is not full-fledged though; provides basic connectivity over Layer4 -- good enough. It does have its own DHCP client as well.
There is a open-source library for ESP8266 for Arduino:
This is what I used, except for few fixes that I had to do. Some of the AT commands were differently used in the library.
Refer to ESP8266_WiFi_Module_Quick_Start_Guide for more up-to-date AT commands.
Here is the video showing my project in action :) Arduino joins my home Wifi network; shows its own IP on the LCD; runs a TCP server at port 1555 and waits to accept new connections. I connect from my desktop Ubuntu and send a message which shows up correctly on the Arduino LCD.
This is going to be a cool extension to my geyser control project now -- I should be able to control my geyser from anywhere and more importantly time it based on the needs. Some sort of security needs to be worked out.