Analytics, Strategy, and Agriculture

Tag: data

Top 5 Considerations for IoT Sensors in Farming

Sensor-based data collection in farming using data collection sensors (Referred to as IoT or “Internet of Things” devices) is growing in all types of crops, as more and more AgTech companies release products to track everything from irrigation levels to crop vigor. In previous posts, I’ve talked about the importance of tracking data to make better decisions on the farm.[LINK] Like with any new technology though, it’s sometimes hard to evaluate its effectiveness. 

  • Did that thing I buy really help the bottom line, or was it something else I did? 
  • How much better was it than the one that cost half as much? 

Along with that, determining what to consider up front before fully committing capital to implement and scale the technology can also be tricky.

Based on working with several sensor vendors over the past several years, I’ve come to view every new AgTech solution through a set of five criteria, or considerations.  

1. Concentration & Cost

When thinking about rolling out sensor-based systems across an orchard or field, it’s important to consider what the ideal level of concentration would be and the cost that goes with that. If a weather station is being sold as the means to track data for micro-climates, how many weather stations do I need per acre? If the real benefit comes from one station per acre or block for example, what will the benefit be if I roll the solutions out over time? Will I get value from just one sensor to start? Or is there a minimal number of devices I need to get any kind of meaningful output out of the devices? It’s important to start slow with any new approach towards farming, but knowing the minimal required concentration to be successful, along with tracking the return on investment over time, is important in justifying the upfront time and money. If it takes ten devices to start, how long till I see results that could justify buying the next ten? And what does that benefit really look like at the end of the day? 

2. Connectivity 

Most sensors require either wifi or a cellular signal to transmit data, so it’s important you have the necessary connectivity across the entire area you plan to deploy sensors, both your initial and eventual planned area. It’s important to know upfront if you need additional fixed wireless towers, a mesh wifi network, or sign up for a cellular plan on top of purchasing and deploying the sensors. 

3. Utilization of Data

So you have installed the sensors, and now they’re connected and tracking information. What do you plan on using the data for? Is it mildly interesting information, or something that will really drive impact to your yield? Some people will track steps or calories with a FitBit, but interest will drop off after 4 weeks. It’s important to make sure your implementation won’t suffer the same fate. Thinking through the use cases for what to use the data for is important to maximize the value of what you’re deploying. It’s also important to discuss with the vendor how that data can be made available. 

Oftentimes, there will be a web-based interface with fixed reporting provided, depending on the sensors. However, discussing means to collect the data (ie connecting to the data in an application, and extracting it) is important if you want to perform your own analysis on the data outside of the fixed set of metrics provided. 

Furthermore, combining that data with other types of data you’re collecting is useful to determine the holistic state of a given acre vs having to bounce between multiple systems to get a full picture of what’s going on. 

4. Scalability 

I touched on this earlier in #1, but you want to get a clear idea of what it would take to scale the sensor system across your entire orchard in terms of cost, connectivity, etc. If you find your initial sensors are useful, make sure you have a clear idea on the roadblocks in front of you, if any, to continue buying and installing more sensors across your farm. 

5. Diversification 

Your final consideration involves identifying what different types of sensors you’ll need to capture a full picture of what’s happening. If you are just investing in soil moisture for the time being, will you eventually need a weather station nearby to track evapotranspiration? If you are using sensors to collect / track bug counts, will you also need to track plant vigor & growth to determine the impact of pest infestations? 

It’s certainly not required to track everything at once, but having a good idea on the types of things you want to track will help you assess what sensor combinations would make the most sense. It will also help you understand what vendors either provide multiple sensors or work together seamlessly to help give you more of a complete picture. 

If you’re only tracking one thing today, are you going with an option that’ll easily let you expand into other things later on? Or will you be stuck using / supporting several different point solutions and trying to make sense of the data yourself across all these platforms? APIs become an important ingredient to extract and combine data for a given acre. 

Though there may be additional things to consider when rolling out sensor-based solutions in your farm, these five points will help you get a clear idea of what you’re getting into and how to maximize the value of your investment. 

If you have any questions about sensors, or IoT, you can leave a message for me under the CONTACT ME above and I’d be happy to discuss your specific scenario in greater detail.

The 4 Parts to a Data-Driven Farm

Data collection, and the tools and services that allow it, provide farmers with possibly the greatest technology benefit across a farming operation. Like calorie counting can help folks lose weight by simply gaining better visibility over what they are eating, data collection can reveal where resources such as labor and capital are being spent to better optimize. This allows for cost savings and greater yields as growers move closer to precision agriculture.

Regardless of the type of crop, farming inherently has always been a data-driven operation. The methods used to collect and analyze data, however, have been changing at a rapid pace. As a result, growers have started to go from clipboards and Excel spreadsheets to ‘Internet of Things’ data collection and business intelligence. Row crops have certainly led the way on automated data collection, given the large number of acres per farm compared to other types of crops. Specialty crops, however, have been catching up in the past couple of years with new AgTech companies forming that focus on crops such as wine grapes and tree fruit.

Where To Start

Before jumping into the mess of AgTech vendors though, it’s important to identify the four pieces you’re going to need to modernize data collection and analysis across your agriculture operations. This will help ensure you have a strong foundation from which to build from.

My advice is to start with data analysis as the first step (even though it’s listed at the end of this post) in order to understand what you want to measure. It’s easy to incur a lot of costs quickly if you bite off too much at once when it comes to building up a modern day data ecosystem. Having a clear set of metrics will allow you to tie what you’re measuring to what you’re saving / gaining and get to a return on investment faster. From there, you can add to what you’ve built, and measure more as your use of data grows across the operation.

Part 1) Data Collection

Recently there has been a surge in sensors for farms that can track everything from soil moisture to pest counts. There’s two things to focus on when considering what tools to use for data collection. 

  1. How many sensors and what type are needed for collection.
  2. How “friendly” the sensors are to feeding data to the system you’ve chosen to consolidate and analyze it in. 

Most vendors offer web-based solutions for showing you their data, but it’s only really useful if you’re able to extract it into a central place (such as a data warehouse) because the real value of this data isn’t what’s happening to an acre of crops by itself, but what collectively is happening within that acre. This is important because you will need to pull together individual readings over a given acre to then compare to an output metric such as yield. As for concentration, without sensors the data is likely already being collected today using manual sampling methods. It’s important, then, to consider the ROI from automated data collection at a sufficient concentration. And of course, always start with just a couple sensors to validate the technology will work for you before scaling. If you only have 1 type of sensor though, to track something specific, then the vendor provided solution may end up working fine.

Part 2) Data Consolidation (or ELT)

Once you have your data collection sorted out, and you’ve begun to measure data in an automated way, you’re going to need to get the data into a central spot, such as a data warehouse, in order to put that data to best use. In that case, you’re probably going to need to pick a vendor to help with this. Though there are ways to do it yourself, it requires hiring one or more experienced data engineers / programmers that can build and maintain the system.

For most farming operations, picking an ELT (extract-load-transform) vendor is going to be a better bet, such as FiveTran or Dell Boomi. These tools will allow you to connect to a data source, such as your accounting system or field data collection, and extract / load / transform the data into a central place which then allows you access to all of your raw data across as many systems as you’re able to connect to. This is why, when considering data collection, you need to understand what vendors have capabilities such as an API which is a capability of a software platform that allows customers to connect to the platform and extract (and in some cases write) data.

Part 3) Data Warehousing

Having a central location for your data is the next foundational component. Depending on the volume of data, you may be able to use a standard database such as MySQL or SQL Server. If the volume of data you’re receiving is large enough (500 GB or more to start with) you may need to consider a Columnar based database such as Snowflake or Azure Synapse which is designed for large data sets. When you’re storing a lot of data from sensors across your operation, you are likely going to get larger data volumes sooner than later. The amount of data a sensor puts out varies quite a bit, so be sure and discuss with your vendor before signing up for the sensors. The key though, is to start with smaller data sets that are useful right away vs starting big by storing everything and having to support too much at once. You will be paying for storage monthly, so having too much unused data will just drive up the cost vs data you can put to work to justify what you’re investing (keep in mind though, storage is relatively cheap in the cloud so it takes a lot of data to drive up the cost of storage). 

It’s also recommended to host your data in the cloud, and to get as much of your ecosystem into the cloud as possible. Though you likely have data hosted locally today, which in some cases is required for business continuity, having a plan to get as much into the cloud as possible has a number of benefits. These include automated back-ups, easy scalability (you pay for what you use, and can increase the size of your server as needed), and streamlined pricing along with the convenience of having the services managed by the cloud provider vs handling onsite servers yourself among several other benefits.

Part 4) Data Analysis

Once you are measuring the right information, and then extracting / storing it in a central location, you need to consider what metrics you are going to track. Having ten or so metrics from which to measure your biggest cost drivers (such as labor or parts) means you can get actionable insights faster, versus guessing what might be useful. It will also help you understand what data to collect first, instead of boiling the ocean and gathering all the data at once. Using a business intelligence tool such as Tableau or PowerBI will allow you to point to your data warehouse and begin to construct these metrics to begin building automated reporting.

It’s recommended to work with an experienced IT resource that has worked with these tools in the past to get your reporting up and running, although if you’re technically inclined and on a budget, there are great BI tutorials available online. Once you have these metrics established, you can look at other capabilities under categories such as data science, artificial intelligence, or machine learning. However, that will require specialized expertise to build solutions using these advanced methods. The good news is these capabilities have been growing over the years, so it’s a lot easier to find expertise with these skills than it used to be a couple years ago. And the tools to roll your own advanced analytics system are getting better every day. 

Next Steps

Though this is a very simplified view of a data ecosystem for your farming operation, I hope this provides you with the 30,000 foot view of the pieces essential to a modern data-driven architecture. Understanding where AgTech vendors fit is critical to ensure you don’t have a bunch of isolated (or “silo’ed”) point solutions that you aren’t able to bring together into a single set of metrics.

The next step is to learn more about each, which I’ll be covering in future blog entries. Of course, if you have any questions, feel free to reach out directly or leave a comment.

3 Steps to take Data Analysis from “Well, That’s Interesting” to Revenue Impacting

Data is hot right now, and it seems everywhere in business these days there is another tool or framework on how to leverage data to drive a real difference in your business.

A good metaphor to understanding data effectively though, is training for an Olympic event (great timing for the metaphor, eh?). Most athletes train, not for the sake of training, but most likely because they want to stand on the podium with a medal as a statement to how good they are at the event. Anyone can train for the sake of training, but competing and winning is really the whole culmination of 4 years of tireless preparation and training.

The same is true for data visualization, in that the real goal should not be building dashboards and pretty charts, but pointing to the direct impact that data had on driving a top or bottom line impact on the company’s revenues. Yes, in larger companies it’s very hard to make a difference on the overall number, but every piece of data should tie to some positive contribution or else what is the point?

Yet, with so much data being made available, and so many people learning how to mine data for insights, there’s a lot of very pretty pictures out there which don’t move the needle at all. Yes, it’s great to hear about athlete’s and how they train, but the credibility isn’t there to the same degree as it is if they’e wearing an Olympic medal. If you’re a budding data analyst, or seasons chart builder, imagine if everything you built had a number next to it that said “this piece of data drove this quantifiable business impact”.

Though data visualization can serve qualitative benefits, such as monitoring a key business process or helping change a perspective on a topic, there should still be some way to tie even those things back to the difference it made (or could make) on the business.

Here’s three steps then, to help that mindset along

1. Understand the Reliability and Structure of the Data You’re Using 

All because you have data, and have access to data, doesn’t mean it’s useful or all that important. You having access to log files from a server, which spits out information on usage patterns of your e-commerce system throughout the day only matters if you’re able to impact that usage in some meaningful way, then tie it back to a positive revenue lift. More importantly though, you have to know the data is reliable and understand how to model it in a way that it produces accurate conclusions. Build some baseline metrics, and measure against numbers you know are correct before going into any complex modeling exercise. Once you put a chart in a slide, it’s out there. So make sure you’re starting from the right data set to begin with. 

2. Develop a Series of Hypothesis about Your Business

Once you know the data you’re working with, and have a good sense of how reliable it is, think about the business as it relates to parts you can actually control / influence. If you’re in advertising, don’t focus on product improvements. If you’re in product development, don’t worry about retention patterns on the website. Think about 3-5 gut instincts you have about how the business could operate differently, then use the data to test out those theories. Don’t simply mine the data, hoping the magical insights just pop out at you. Data, like a car, helps you get to where you’re going – it won’t take you there on it’s own. You need to at least have a rough idea of what you’re looking for, so you can build worthwhile visualization to help vet that hypothesis into an actual conclusion.

3. Focus on Low Hanging KPIs, then Expand from There 

It’s easy today, with technology making more data accessible with less effort, to try and go after ground breaking insights. However, you no doubt have areas of your business you can directly impact and know will make a sizable difference to the bottom line, that you need help in influencing. Start with thinking through 3-5 KPIs coming from your hypothesis that you can build from the data you’ve vetted, to influence key business people or back up your assertions with a new direction you’re moving your team in.

If you can measure it, you can prove it (as long as it’s reliable and true), but remember that it’s never a silver bullet or the whole story. Data can be powerful, but it can also lead people astray or get people focused on the wrong things. When done right though, using validated data tied to a hypothesis and measured in a way that drives a meaningful revenue outcome, it can make a big difference.

Why Tableau is Worth Billions: A Case Study on Becoming a Digital “Port City”

It’s appropriate that Tableau is located in Seattle, as they both became popular for similar reasons.

Seattle, started as a logging town shipping lumber down to San Francisco, then hit a big boom during the Klondike gold rush followed by a big shipping boom. It then moved into a big boom in aerospace followed by the growing influence of technology – starting with Microsoft. Access to resources, and a connector between multiple places. Seattle was big on logging, because there was an ocean that made it easier to transport lumber south, with the means to make it accessible and useful. Seattle was big during the Klondike gold rush, because you could take a ship from Seattle over to Alaska, and provided the resources and shipping to get there. Seattle was big into aerospace, because William Boeing got things kicked off here so there was the resources and buyers to set up a shop and build an aerospace business. Seattle then became big into technology, because Bill Gates changed the world with Microsoft Windows and people could come and leverage the resources that created.

Now lets look at Tableau – From wikipedia: “In 2003, Tableau was spun out of Stanford [9] with an eponymous software application. The product queries relational databases, cubes, cloud databases, and spreadsheets and then generates a number of graph types that can be combined into dashboards and shared over a computer network or the internet. The founders moved the company to Seattle, Washington in October, 2003, where it remains headquartered today” 

Tableau then, wasn’t famous because it invented data or created a better way to store data. Rather, the platform made that “digital lumber” we know as data more accessible. It became a way for an average user to reach out into the data space and extract useful information, which they can then use. In effect, Tableau is the digital “port” city for many business owners, that provides access to that raw material and the capability to make it useful.

Becoming a digital port city then, isn’t all about what the platform provides in and of itself but the material it helps you gather / process / leverage. Social media is billions of messages, but Adobe’s Marketing Cloud promises to make quantifiable sense of it all. Server log files are completely useless in and of themselves, but Splunk helps turn all that into a meaningful dashboard.

Lots of tools exist out there, promising to mine assets and turn them into something useful. But as data became a boom, and the trend grew, you could also see the rise of companies like Tableau growing along with that tide. If cities in the 1800’s decided to use clay instead of lumber, perhaps Seattle may never have taken off.

What’s important to note then, is that becoming a digital port city can produce a tremendous amount of value as long as the resource you’re accessing is growing in popularity. However, everything (even data) only stays a popular trend for so long. The hope is, then, that you’ve grown enough to sustain yourself until the next wave takes off and you can successfully adapt along with it. Tableau is in it’s first major boom cycle, as Seattle grew with lumber. As history has shown though, Seattle had many boom and bust cycles as time goes on. How many companies also rise and fall within a single hype cycle (ex: Detroit) ?

Becoming a digital “port city” and staying that way really comes down to 3 things

1. Don’t oversell the hype (to yourselves or your clients)

No matter how on fire your company might be today, every marketing pitch or slogan only has so much gas in the tank. Focusing on the broader industry issue (ex: revenue growth vs access to data) means you’ll continue to stay relevant long after the initial hype has passed. Take advantage of a trend’s popularity, but don’t so closely associate yourself to that one thing that you can’t exist without it – what if Kodak had focused on better living through chemistry vs film? As film declined, chemistry surely didn’t go out of style. And as it turned out, Kodak had some of the most talented chemists in the world working for them because film is a hard thing to make. What would have changed, if Kodak’s brand became focused around something that wouldn’t ever go out of style, vs a single product focus? 

2. You’ll have to think of the post-hype at some point 

Yes, it’s important to stay hyper-focused on your core competency and capability during a big sales cycle, but long term planning focusing on “what do we do when people don’t care about X trend any more” is important. Google will have to figure out ways to make money, after online advertising. Facebook may not be the hot social network 100 years from now it is today, and Microsoft is already starting to evolve in a world that cares less about personal computers. Tableau, too, has the talent and revenue to think about what’s next in the data space long after people stop caring about 2D data visualization in the form of accessible dashboards. Though we have examples, every company has to overcome it’s own culture and leadership challenges to continue to evolve and adapt. 

3. Build a foundation around the longer term trend, while capitalizing on the current hype 

Say you’re Boeing, and you’re contemplating life after airplanes, or perhaps investments that build a platform of services focused on a single brand element of your company. Do you diversify, by extending your reach into other areas of aerospace, or do you step back and say “well, our real purpose is to connect people, so lets invest in other ways to connect people outside of just flying them together”. It’s a tricky question, with no easy answer, which could mean botched acquisitions and a confusing marketing plan if you’re too broadly focused. However, tying in telepresence as part of the “connecting people together” strategy may mean infrastructure investments in aerospace communications networks, that you wouldn’t otherwise make, to allow video chats in airplanes while investing in smaller start-ups that focus on video codecs and compression algorithms that might net you a decent return down the line.

Focusing on just building airplanes though, Boeing would never invest in a Skype, but down the line will it be too entrenched to see a decline in aerospace with the will to shift their focus? Skype would have been a bad idea for Boeing, but what about investments in technologies that make it easier to transmit video which is entirely something they could leverage today? It’s not easy to do, and a lot of companies get it wrong here, but focusing your core message and internal alignment on something bigger than the immediate trend or fad is important, if you want to build a company that’ll be around 30+ years down the road.

If you do those three things successfully, whether you’re a city near the Ocean or a data analysis tool helping unlock value, you’ll no doubt continue to justify the value you bring long after that initial wave has past. It’s why Seattle continues to thrive, whereas cities like Detroit have struggled, and why Tableau is worth billions as a tool that accessed data without developing/ hosting/ managing most of the backend infrastructure that makes up those data systems. Stay beholden to only one path, or one product and you could go from the top of the pile to getting buried by your competitors. Toyota would say it’s not a car company, but a transportation company – because cars are only relevant for a period of time, but people will always need a way to get transported.

Become a digital “port city” by making a useful resource accessible and useful, then focus on continually evolving as the thing people need access to changes.

3 Reasons Why You Need a Chief Analytics Officer

Data has exploded in a way that rivals mobile’s explosion ten years ago. Everyone is out there buying masters degrees, data visualization licenses, and data scientists by the truck loads in a way that mimics corporations buying mac laptops, mobile developers, and app store branding when iPhones blew up the smart phone space.

The Analytics ‘Trend’ Isn’t New

There are a lot of great things taking place right now with all the interest around data analysis, but the funny thing is that data analysis is nothing new (neither is data science). There’s a good 30-40 years of work on data, from data architecture to database administration (not to mention the millions of excel spreadsheets that corporations are running critical business functions on) that live inside companies and create a legacy layer that this latest wave of data analysis is building on.

Other new trends, such as big data analysis and the cloud computing revolution, have further spurred companies to consider ways to extract usefulness from their existing data and move away from churn or ARPU and develop distinctly competitive analysis with phrases like “regression analysis” and “predictive analytics” becoming much more common in corporate board rooms.

Translating Data

The big problem is, as was the case with mobile, is that you have to be able to translate interesting technology into impacting ROI-laden investments that drive top or bottom line revenues (or create efficiency and lower costs of course, as well). There’s a good deal of buzz around big data being an overused term, and a hundreds of millions of dollars spent on visualization tools will, at some point, taper off when the average business user turned dashboard builder runs out of things to visualize due to saturation, bad data, etc.

So Who / What Is This Chief Analytics Officer?

A Chief Analytics Officer could be a Director of Data, or a VP of Analytics, but having someone at an executive level that can drive a centralized data strategy for the company should exist for these three reasons.

  1. Centralizing Your Data Resources Will Help Avoid Silo’ed Capabilities

To turn all this hype into profit, it means building a centralized capacity. A capacity which sites outside of the IT-to-business politics and hype to buy visualization tools, and instead focusing on building a stack of capabilities, from the data lake to the dashboards, geared around revenue generating use cases taken from business partners who need more usefulness from their data without having to build silo’ed data science teams that rely on fractured data sets.

When anything is this pumped up, every department is going to want to get involved and build capabilities, since every business group uses data in some form or another. The problem is that it takes a variety of experiences and backgrounds, along with investments, that need to be built at a corporate level with a plan to centralize some capabilities and decentralize others with a clear data strategy that everyone can get behind.

Centralizing this capability means one strategy, one leader, and limitless opportunities for everyone to participate without each department deciding their own game plan for riding this data wave.

  1. Consolidating data to maximize usefulness, while aligning that effort under a single leader

The topics around big data, and data lakes are growing overwhelming, with more and more companies working to consolidate all their data in one place to allow for both advanced analytics & traditional business intelligence functions. At the same time, a data lake built in the wrong way can cause latency along with too many executive peers building extensive requirements which ultimately brings any progress to a halt.

Bringing your data consolidation effort under a single leader, tied to a data strategy that brings the bigger outcomes into focus and alignment while leaving the smaller day to day details up to a single org unit means your company can spend less time planning & debating, and more time driving value from your data lake.

  1. Impact is prioritized, over ‘interesting trends’

Much like the millions of dollars spent on corporate mobile apps that never got traction, companies today are spending millions of dollars on real time streaming, data visualization, and corporate education on DAX programming all in an attempt to capitalize on the data analytics hype and create a stronger bottom and/or top line revenue stream through the use of data analysis.

The thing is, data isn’t a new domain for technology, nor is investing in Big data going to revolutionize your company.

There’s a good deal of effort being spent on building impressive looking visuals, which add no incremental value over the same data displayed in an excel chart. Furthermore, companies investing in hiring legions of data scientists without clear revenue-driving hypothesis will find they spend a good deal of time figuring out just what to focus on.

As is the case with any over-hyped technology, whether it’s enterprise wide tableau licensing or infrastructure to support web traffic analysis for real time personalization, the tools are only as good as the capabilities on the team and the business cases they are actively working towards.

Focusing on a single leadership structure to come up with the real tangible value for investment in data analytics means there’s a common set of goals that’s driving the spend, and a clear idea of what each department and employee is focusing on.

It’s not so much that a single team owns every analyst, but rather each instrument is calibrated so the whole company sounds like a beautiful concerto vs a number of instruments playing at different rhythms.

Furthermore, when it comes to the vendor onslaught and procurement nightmares that naturally arise in the midst of a technology boom, there’s a clear investment strategy for how the company plans to leverage capabilities such as big data or advanced analytics. This can influence everything from recruiting and training, to infrastructure and software licensing, and help ensure each investment is additive vs expensive and lacking in impact.

There’s a good deal of interesting happenings in the data space right now, but companies need more impact to back up the cost.

There are no doubt other benefits I’ve missed out on taking data seriously, and putting someone in charge who is somewhat removed from the politics and inefficiencies that come from burying the capability inside an existing org (similar to the CIO coming of age, and now no longer reporting to CFOs in most companies).

The aim is however, to ensure your data analytics efforts are making a meaningful impact, and driving the kinds of returns most companies never experienced during the mobile app boom almost ten years ago now. And in so doing, benefiting every company that invests in the great capabilities a data-driven org has at its disposal.

3 Ways to Make Data Visualization Useful

Data visualization, the front end of data analysis that makes everything more understandable, interactive, and attractive, is taking off like never before. Many companies have risen to helping establish accessible data tools that are both easy to operate, and understand to where anyone today can point the tool at a data sent in particular and begin developing dashboards, and charts like never before.

However, democratizing data analysis and working towards creating self-service BI can have very negative implications if the discipline and rigor behind that data analysis isn’t handled along with developing all those great dashboards. Furthermore, you can easily find yourself building interesting charts without being able to articulate why those charts are impactful, and what knowledge you’d need to back up if something irregular pops up that someone calls out in a meeting.

People would agree, for the most part, you shouldn’t take someone off the street that isn’t an engineer and have them design & build an airplane. Even if the tools to design an airplane become accessible, engineering is important to make sure the plane flies successfully when it’s actually constructed and launched.

The same should be true for data, though most data isn’t related to life or death situations, there is data misused or incorrectly calculated that can bring a company to it’s knees – from bad sales figures, to bad market analysis resulting in tweets that spin the company into damage control.

With that in mind, I have 3 steps to make your data visualizations more accurate and useful, so your understanding of data can go hand in hand with your energy to leverage it.

1. Understand Data Modeling Fundimentals 

This may sound / seem like overkill, but if you’re going to work with a tool like Tableau or Qlikview, having the basics down around how data works and how to model it effectively means you can go into some data source somewhere, understand it down to the elements themselves, and join that data together in a way that allows for meaningful and accurate analytics.

If you don’t know what an inner vs outer join is, then you’ll have a hard time even pulling together the data into a tool like Tableau without potentially impacting the outcome.

The best book to dive into here is “The Data Warehouse Toolkit” By Ralph Kimball (Kimball is the godfather of dimensional modeling, and is used by most all BI people to develop “cubes” for data analysis). http://www.amazon.com/Data-Warehouse-Toolkit-Definitive-Dimensional/dp/1118530802/ref=sr_1_1?ie=UTF8&qid=1443215078&sr=8-1&keywords=data+warehousing

It’s a hard thing to get through, especially if you want to stay out of the weeds of data management, but it’ll get you deep enough in the fundamentals around good data governance and management, that you’ll be far more effective at building compelling and accurate data models.

2. Focus on Impact vs Interesting

The world is full of interesting data, that would make for all kinds of interesting conversations. However, very little of that translates into impactful data that can make a material impact on a company’s bottom line. Knowing what’s interesting vs impactful can make the difference between a bunch of nice looking visualizations, vs an impacting dashboard that drives business change and makes what you’re doing both useful and practical.

There are a lot of books out there that show examples of data visualization, and the majority are certainly interesting and informative. However, if they don’t have a direct impact on helping change your business in some way, then you might as well frame and hang those pictures on a wall. Develop a clear hypothesis, know what you’re looking to get from the data, and work the problem through to a conclusion.

Data journalism is a great approach towards this, that combines story telling with a clear impact, call to action, or outcome.

A good resource for where to begin is at http://datajournalismhandbook.org/1.0/en/

3. Know your audience

The most important step in making data visualization useful is to know your audience, and tailor the output in a way that makes it the most useful for the consumer. A CFO typically won’t want to see the same visualization as a CMO, and with the majority of data visualization tools allowing for the ability to filter / slice / drill based on the data available in a cube, you can tailor content like never before and peform ad-hoc data analysis with less construction up front required.

Understand what questions your audience might ask ahead of time, and consider how your material is fluid enough to respond in turn. You might be building a dashboard for showing sales nationwide for your company, but what if they ask for one product vs another? Can you build your data model to support that, then add a filter or will you have to go back, work for a week, and bring the specific chart back?

There’s tons of great data points out there, waiting to be discovered and shared via data visualization platforms to help enhance and enlighten business users at all levels of the organization. Make sure though, before jumping into the fray, that you have the foundation, direction, and foresight to develop something meaningful.

Don’t Go on a Date with Data, Marry It

There are a lot of companies out there today sitting on several types of data, with various levels of integration. For most, it’s a process involving multiple individuals to extract, transform, and develop that information into a useful dashboard or decision-support system given the company has legacy infrastructure and several different third party vendor platforms all producing useful information but it’s in own specific format.

The big challenge then, that companies face today, is how to tie all of this information together in a way that provides meaning but more importantly, provide it in a way that can be near real time with the ability to tie different types of data from all over the enterprise together.

Yes, you know how much you lost and earned in a given period of time, but can you determine why with the ability to stitch together different data sources? Can you do your own regression analysis, or perform correlation research to determine what trends might be causing a rise or decline in monthly revenues? Or do you have to hire it out, and wait several weeks, while analysts inside or outside your company work to produce the answer to that one simple question?

There are lots of people willing to charge you for the ability to do this month to month, on an ongoing basis, but if you’re content with having systems kept in disarray while you’re driving up the cost of labor to mine it every time you have a question about your numbers, then you’re simply dating your data.

You’re content leaving data at arm’s length, getting to know it to a point, but you’re not spending the time and effort it takes to really get to know the data inside your company and make the investment to make data a key part of your professional life. Marriage means accepting the ugly truth about someone, and acknowledging to them the same sometimes ugly truths as well, to build a close relationship.

Marrying data means you’re accepting the truth that legacy infrastructure, silo’ed data sets, and weeks spent building a single dashboard isn’t working for you and you’re ready to spend the time and effort it takes to bring data closer to the core of your business. It means making the investment to build an agile analytics platform that allows for ad-hoc analysis, and spending the time it takes to get your leadership team on board with understanding what a regression is and what level they feel comfortable being trained to help drive insights without an army of external analysts.

Marrying data also means accepting the things you can’t change about it, and learning to live with limitations as they are. It doesn’t make sense to send everyone to school to learn data science, but it does make sense to get up to speed on what it means and getting people trained in the vocabulary of data so that those highly trained resources can build what is needed and make sure everything you’re investing in has a clear return at the other end.

Making the ultimate commitment to data, vs having an off again on again relationship, means you’re wiling to spend the time it takes to make the upfront investment to clean up the silos and tie together the systems keeping your really valuable insights locked up. It means knowing that it’ll take time to see the value, but it’ll be worth the investment, vs continuing to grow your OpEx budget on consultants and FTEs working with what’s there now, and spending time sorting and tying each system together for one-off requests.

More importantly, marrying data means you’ve accepted that the key to a happy company is a happy data warehouse, and that Innovation in it’s most meaningful way means you’re able to draw out from your past what solutions and ideas might help fuel your future. By spending the time and effort to build meaningful data interconnectivity, along with the systems necessary to analyze and understand that data, you’ll be able to see what trends are coming your way and how you can be proactive to meet the challenges in an ever changing industry environment.

You’ll reduce the risk of being disruptive, you’ll be armed with answers before the questions get asked, and you’ll be able to walk hand in hand with your data into the sunset while every division within your company gets insights they need to better track what’s working and what isn’t along with driving new revenue streams to your customers.

A happy marriage for some seems like a fairy tale, and it’s not going to solve every problem you encounter. However, if you’re willing to put in the time it takes and accept that there’s things you could be doing to pay better attention to your data along with spending the time to care of it, your data will produce insights and become more open to analysis as a result which will always work better than keeping it in a chaotic state and spending time doing one-off reports.

So consider what marrying your company data looks like for you, and build a plan and a roadmap to make data more meaningful and acceptable for your company. I can guarantee you, your competitors are probably already doing the same.

Dan Maycock is the author of “Building The Expo”, which shares best practices on leveraging #Innovation in meaningful ways and saving the concept from it’s overused but underutilized past. The book has first hand stories, and best practices from Dan’s years of experience working with Fortune 1000 companies dealing with emerging technology adoption in an increasingly dynamic business environment. You can purchase the book atAmazon.com or learn more about Dan at http://www.transform.digital

4 Ways to Effectively Use Data In Your Job

With all the excitement around how companies are using data today, it’s hard for anyone outside of a job specifically dealing with data to know how to effectively use it for their day to day work. Yet, there isn’t a single career that isn’t impacted by the use and understanding of data, and the more effective someone becomes at harnessing and understanding data mining, the more they can impact the things that impact their professional ecosystem.

From impacting your online brand, to better tracking variables you deal with around a given task at work, knowing how to leverage data can make a big difference in any number of careers.

1. Start with a question

Before diving into a number of articles or tools regarding data, start with the question of what you’re trying to answer. It sounds basic, but you’d be surprised how often I’ve worked with people that have said data is the answer without first having the question. Figure out what are the most pressing business problems you, your boss, or your company are facing and see how data might help provide insights to answering those important need to knows.

2. Start with a small amount of data, build from there

It doesn’t take petabytes of data to answer questions, sometimes it can be a relatively small set of data to answer big questions. With all the hype around big data, sometimes it’s hard to realize that with only 100 or so records, and a pivot chart, you can get to important answers that are far more useful than what a million records could show, depending on the type of data and the question you’re looking to solve.

3. Leverage third party data that’s free

There’s a TON of data out there that’s completely free, and useful to use. US Census is a great place to start, and there are a number of sites, such as Google’s public data directory that’ll let you explore it. Furthermore, you can download the data for free and combine it with your own internal data to add greater context for things like taking your company’s store sales by zip and seeing how demographic trends within those zip codes may impact certain buying habits.

4. Learn about Data Mining 

The key to making data useful is by learning methods that allow you to tap into data, and find useful data points that can help solve the business problems you’re looking to tackle. Data mining is the practice that helps you start to uncover trends and patterns in data, and is a great discipline to begin with, whether it’s using Excel and a little bit of data or tapping into RapidMiner and starting to dive into Hadoop, Data mining spans the gambit on complexity and data quantities. Remember the first three points to keep the right context and not go overboard too soon though, and you’ll be in good shape.

Regardless of your career, there is a way data can no doubt help you professionally and impress your co workers and higher ups in the process. Start with the fundamentals, help answer important questions, and simply build from there and you’ll be a bonafide data analyst before you know it.

Why Everyone Should be a Data Miner

In thinking about the topic of data mining, a lot of different types of roles pop up in people’s minds. From data scientists typing away in giant data centers, to DBAs sitting in cubicles processing large amounts of corporate data, to an analyst building a spreadsheet for an annual report contribution.

Maybe it’s something far more physical, bringing up images of pick axes and hard hats and a big block of data (however that’s visualized, probably with 1’s and 0’s – all matrix like). Regardless of the image that comes to mind, it’s probably hard to fathom every business professional in some form or another becoming adept at data mining, and considering it a critical competency to keep in their professional toolbox in the years to come. Yet, when we explore the topic, we can easily see how data mining could become one of the preeminent skills that set folks apart in an era where it’s harder and harder to stand out from an increasingly noisy and competitive work climate. Lets start by looking at the six attributes that make up data mining (as defined by Wikipedia)

  • Anomaly detection (Outlier/change/deviation detection) – The identification of unusual data records, that might be interesting or data errors that require further investigation.
  • Association rule learning (Dependency modeling) – Searches for relationships between variables. This is sometimes referred to as market basket analysis.
  • Clustering – is the task of discovering groups and structures in the data that are in some way or another “similar”, without using known structures in the data.
  • Classification – is the task of generalizing known structure to apply to new data. For example, an e-mail program might attempt to classify an e-mail as “legitimate” or as “spam”.
  • Regression – attempts to find a function which models the data with the least error.
  • Summarization – providing a more compact representation of the data set, including visualization and report generation.

Though the definitions seem somewhat dense, think about how you’d be able to take any job – from being able to use regression analysis to construct a real estate data model to improve pricing predictions, to using summarization to build a better financial report for your senior leaders to interpret how great of a quarter you had.

Though some methods of data mining are harder than others, and you can quickly get in way over your skis without proper learning, knowing how to sift through data, and pull out the useful stuff, will give you a greater sense of the world you work in by understanding the data that matters and it’s so easy these days to learn data mining techniques online!

Just typing in “data mining classes online” produces hundreds of leads, from Coursera to MIT open courseware. Though some options go into areas like Data Science, which is much deeper level analysis, it all starts with understanding data and how best to derive meaning from it – regardless of how deep into the weeds you want to go.

This in turn gives you a big foot up against your competitors, who are largely relying on other services / people to hand them processed data and conclusions to do something with. Going from a commodity to a distinct competitive advantage means going in a direction others aren’t, and just having a nicely worded dictionary isn’t enough these days – you need to be able to turn that dictionary into a novel, and tell a story with the data that will reveal things about your business or your industry that’ll drive better decisions through unique insights.

Does Your Company Have a Chief Data Evangelist?

A lot of companies are talking about Chief Data Officers, but what about having a chief data evangelist instead?

Recently I was talking to a good friend of mine that works in the Business Intelligence space about the concept of a Chief Data Officer being brought up in the halls of different companies around the US (mainly of course, IT departments dealing with the onset of new data solutions to handle all their data.)

What he shared was that companies should focus less on centralizing data to get to a single version of the truth. Instead, they should focus on recruiting a chief data evangelist to get groups within a company on board with a set of standards that they can build data models around for use within their team, then grow grassroots communities within their company. This could be akin to a data “co-op” of sorts which could, in turn, enable teams to take their own data models and share data at a bottom up approach vs simply being drug along by a chief data officer from a top down approach, marching to the beat of centralized data control.

This extreme decentralization has worked in other facets, including executive leadership as characterized in the book “The Outsiders” by William Thorndike so why couldn’t it work with data?

As I began to think about it, it does make sense to have people in your organization advocating for best practices, and getting different groups on board with a set of standards but leaving the usefulness of the data to the teams using it, as no two groups of course ever have the same need for a specific data set in a specific format.

Though larger efforts like data warehousing will remain centralized activities, imagine what companies could achieve through extreme decentralization focused on evangelism of standards and organization level adoption & modeling efforts that in turn drive community activities within a company vs dragging along the enterprise one team at a time to conform to centralized data models that may or may not work for them.

Seems like a much better solution to me. In thinking about what a Chief Data Evangelist might do at your company, consider the following job description

Task #1) Strong understanding of best practices around data governance, data management, and data modeling for the purpose of leveraging corporate data for use by a specific team

Task #2) Desire to get teams within a company on board with leveraging standards for data governance and modeling, for the purpose of collaborating with other teams and sharing data within organizations / company

Task #3) Make a killer salsa

If that sounds like a great job description, perhaps the job is for you. Regardless of who has the role, be it official or unofficial, having strong advocates for standards along with proponents for data / BI communities in your company can go a long way in helping drive greater adoption of data solutions within your company and help grow data-driven solutions in the process.

© 2021 DanMaycock.com

Theme by Anders NorenUp ↑