Bengaluru emerges as global hub of Hadoop engineers
BENGALURU: India is no longer an IT sweatshop. As data becomes the new oil, India is gaining more experts who can handle large volumes of information.
Bengaluru has emerged as a major global hub of engineers trained in Hadoop — an open-source software framework and an operating system of sorts for the data vertical, in demand at companies such as Flipkart that seek to leverage data to gain an edge over the competition.
India’s growing pool of Hadoop experts is also attracting several US-based companies to consider expanding in the country.
“After Silicon Valley, Bangalore has the largest number of people with Hadoop skills. The talent pool is huge,” said Arun Murthy, founder of US-based Hortonworks, one of the earliest adopters of Hadoop. Hortonworks’ platform helps companies implement Hadoop in their data architecture.
“We have a small team of 10 in India. That will soon increase,” said Murthy, 34, who has been working on Hadoop since 2007.
While there are no official records to establish Murthy’s claim, data from online portal Meetup is indicative of this trend. Bengaluru has 2,339 Hadoop enthusiasts, while Silicon Valley has 2,552. Country-wise, India has 6,561 Hadoop users across its four metropolises, and the United States has 8,475 users across New York, Boston, San Diego and San Francisco. Other startup hubs such as Tel Aviv (866), Singapore (852) and Beijing (31) figure lower.
Hadoop was the name of a child’s toy elephant. The father Doug Cutting decided to borrow that name for the open-source software framework he helped create.
Hadoop is used to store and process large volumes of data with the help of disparate computing systems, instead of a few supercomputers. The hardware is cheap and the software is open-source, leveling the field for all players.
With newer versions, Hadoop has emerged as more of an operating system that manages resources, secures data, and performs other functions. Applications written on top of it help process data and cross-pollinate data points.
In addition to Hortonworks, US-based dynamic price optimization tool provider Boomerang Commerce, and sales performance manager Xactly Corp are adding high-skilled workforce in India. Twitter recently announced that it will set up its first research and development centre outside of the United States in India.
The rise in the number of professionals proficient in Hadoop has been fuelled in part by India’s billion-dollar successes such as mobile ad network InMobi and online retailer Flipkart.
For InMobi, which generates 5 terabytes of data per day and processes 240 terabytes of it, Hadoop is a critical component. “We have several teams making contributions to the open source community,” said Rajiv Bhat, vice-president of data sciences and marketplace at InMobi. Last year alone, InMobi’s technology team submitted 10 papers at the annual Hadoop Summit.
“Hadoop is de-facto at Flipkart. We have made significant investment in data intelligence,” said principal architect Sharad Agarwal, who was one of the founding members of Hadoop 2.0 along with Horton works’ Murthy and Cutting.
He regularly contributes code to the community and organizes Hadoop meetups in Bengaluru. “We have had 8-9 meetups so far; more and more people have been attending it,” said Agarwal.
“Now lots of software engineers are choosing this as one of the areas of specialization,” said Dr Shailesh Kumar, a member of the technical staff at Google whose primary focus is using machine Learning, data mining, and computer vision for various Google products.
The Hadoop framework is soon expected to go mainstream: “The remaining minority of dazed and confused CIOs will make Hadoop as priority for 2015,” Mike Gualtieri, principal analyst at Forrester, said in his predictions for 2015.
The Latest Software Testing News department was not involved in the creation of this content.