The oil and gas industries are facing major challenges – the costs of extraction are rising and the turbulent state of international politics adds to the difficulties of exploration and drilling for new reserves. In the face of big problems, its key players are turning to Big Data in the hope of finding innovative solutions to these pressing issues. Big Data is the name used to describe the theory and practice of applying advanced computer analysis to the ever-growing amount of digital information that we can collect and store from the world around us. Over the last few years businesses in every industry have enthusiastically developed data-led strategies for overcoming problems and solving challenges, and the oil and gas industries are no different. Royal Dutch Shell is one of the largest oil and gas companies – one of the “supermajors” which also include BP, Chevron, Total and ExxonMobil – and the world’s fourth largest company by revenue. For some time now it has been developing the idea of the “data-driven oilfield” in an attempt to bring down the cost of drilling for oil – the industry’s major expense. A recent survey by Accenture and Microsoft of oil companies and those involved in the support industries found that 86% to 90% of respondents said that increasing their analytical, mobile and Internet of Things capabilities would increase the value of their business. The search for new hydrocarbon deposits demands a huge amount of materials, manpower and logistics. With drilling a deep water oil well often costing over $100 million, no one wants to be looking in the wrong place. Surveying of potential sites involves monitoring the low frequency seismic waves that move through the earth below us due to tectonic activity. Probes are put into the earth at the spot being surveyed, which will register if the pattern of the waves is distorted as they pass through oil or gas. In the past this would involve taking a few thousand readings during the typical survey of a potential drilling site. But in the past few years technology has advanced to the level where it could involve over a million – vastly increasing the amount of data gathered during exploration. Shell uses fibre optic cables, created in a special partnership with Hewlett-Packard, for these sensors, and data is transferred to its private servers, maintained by Amazon Web Services. This gives a far more accurate image of what lies beneath. Data from any prospective oil field can then be compared alongside that from thousands of others around the world, to enable geologists to make more accurate recommendations about where to drill. Production forecasting is one of the first jobs – determining the likely output of the reservoir is key to determining what resources should be spent on collecting it. When this decision is data-led, operators can have more confidence that this will be done efficiently. Shell also uses Big Data to ensure its machines are working properly and spending as little time as possible offline due to breakdowns and failure. Machinery used in drilling has to operate in harsh conditions for prolonged periods of time so is prone to wear and damage. To counteract this, the machinery is fitted with sensors collecting data about its performance and comparing it with aggregated data, meaning parts can be replaced in an efficient manner and downtime minimized, further reducing overheads. As well as exploration, Big Data is being put to use to streamline the transport, refinement and distribution (retail) of oil and gas. Shell is vertically integrated, and therefore involved in every aspect of the process through to packaging and selling it to the consumer as fuel for their car for heating their home. Refineries have limited capacity and fuel needs to be produced as close as possible to its point of end use, to minimize transportation costs. Complex algorithms take into account the cost of producing the fuel as well as diverse data such as economic indicators and weather patterns to determine demand, allocate resources and set prices at the pumps. Of course, we’ve long been conscious of the fact that we could eventually use up all of the non-renewable oil and gas buried under the earth – perhaps sooner than we think. While this is an environmental concern to us all, it’s a financial one to companies like Shell. Dwindling reserves mean the cost of getting at what is available goes up, as they are forced to look deeper underground in ever more remote locations. One alternative is offered by the growing hope that “unconventional resources”, such as shale gas and tight oil will fill the gap. These resources, trapped in shale and sandstone, now supply 20% of the gas used in the USA and their use is expanding rapidly around the globe. However they demand new extraction methods such as hydraulic fracturing (“fracking”). Because this is relatively new far less data is available so exploration can be more hit-and-miss. However new techniques are being developed to use the data that is being collected from probing and drilling these sites to help pinpoint other locations where reserves could be hiding. All of these elements form the “big picture” – every part of which must be synchronized if a company is going to prosper in the costly, dangerous and highly competitive field of oil and gas production. Thanks to big data analytics, the interconnection between these elements of the business can be examined and monitored in detail. This means models can be built and simulations created by analysts, to explore how minor tweaks to a certain area of operations could have big impacts on the productivity or efficiency of another. The vast amount of data collected from all areas of the company’s activity means the result of the simulations will hopefully be as close as possible to the way things will play out in the real world. Ultimately this leads to decision-makers being better equipped to make the decisions that affect the company’s fortunes.