Reports – The White Pages

Excel is an excellent tool for analyzing data. An analyst can easily connect to and import data, perform analyses, and achieve results quickly. Export to Excel is still one of the most used features of any Business Intelligence tool on the market. The demand for “self-service BI” resulted in a lot of imported data being stored in overly large Excel files. This posed several problems. IT administrators had to deal with storage requirements. Analysts were restricted by the amount of data they could work with, and the proliferation of these “spreadmarts” storing potentially sensitive data created a governance nightmare.

A little history

Power Pivot was created to provide a self-service BI tool that solved these problems. Initially released as an add-in for Excel 2010, it contained a new analytical engine that would soon be introduced to SQL Server Analysis Services as well. Its columnar compression meant that millions of rows of data could be analyzed in Excel and would not require massive amounts of space to store. Data in Power Pivot is read-only and refreshable – ensuring integrity. It allowed analysts to set up their own analytical data sets and analyze them using a familiar looking language (DAX), and visual reporting canvas (PowerView) all from within Excel.

The original version of Power BI brought PowerPivot to Office 365 through Excel before Power BI’s relaunch gave it its own consumption interface (the service) and design client (Power BI Desktop). Both the PowerPivot engine, and Power Query were incorporated into the service and Power BI Desktop, while the Silverlight based Power View was replaced with a more web friendly reporting canvas.

Excel support

Throughout all these changes, Excel has continued to be well supported in the Power BI service. Analyze in Excel allows an analyst to connect to a deployed Power BI dataset (built with Power BI Desktop) and analyze it using pivot tables, charts, etc. Recent “connect to dataset” features have made this even simpler. Organizational Data Types allow Excel data to be decorated with related data in Power BI.

Excel workbooks containing Power Pivot models have always been supported by the service. These models can even be refreshed on a regular basis. If the source data resides on premises, it can even be refreshed through the on-premises data gateway. This all because the data engine in Power BI is essentially Power Pivot.

It’s that word “essentially” that causes a problem.

Datasets that are created and stored within Excel workbooks are functional but can only be accessed by that workbook. Contrast this with a dataset created by Power BI Desktop, which can be accessed by other interactive (pbix) reports, paginated reports, and as mentioned above, by Excel itself. The XMLA endpoint also allows these reports to be accessed by a myriad of third part products. None of this is true for datasets created and stored in Excel.

So why would anyone continue to create models in Excel. The reason has been until now that although Excel can connect to Power BI datasets to perform analysis, those connected workbooks would not be updated when the source dataset changes. This meant that those analysts that really care about Excel needed to work with the Excel created models. This changed recently with an announcement at Microsoft Ignite Spring 2021. In the session Drive a data Culture with Power BI: Vision, Strategy and Roadmap it was announced that very soon, Excel files connected to Power BI datasets will be automatically updated. This removes the last technical reason to continue to use Power Pivot in Excel.

Tooling

Building a dataset with Power BI Desktop is fundamentally the same as building one with Excel. The two core languages and engines (M with Power Query, and DAX with Power Pivot) are equivalent between the two products. The only difference is that the engine versions found in Excel tend to lag those found in Power BI Desktop and the Power BI service itself. I’d argue that the interfaces for performing these transforms, and building the models are far superior in Power BI Desktop. not to mention the third-party add-in capability.

In this “new world” of Excel data analysis, Datasets will be created by using Power BI Desktop, deployed to the service, and then Excel will connect to them to provide deep analysis. These workbooks can then be published to the Power BI service alongside and other interactive or paginated reports for use by analysts. With this new capability, Excel truly resumes its place as a full-fledged first-class citizen in the Power BI space.

What to use when

With this change, the decision of what tool to use can be based completely on its suitability to task, and not on technical limitations. There are distinct types of reports, and different sorts of users. The choice of what to use when can now be based completely on these factors. The common element among them all is the dataset.

With respect to report usage, typical usage can be seen below.

Tool	Used by	Purpose
Power BI Service	Report consumers	Consuming all types of reports: interactive, paginated and Excel
Excel Online	Report consumers	Consuming Excel reports from SharePoint, Teams, or the Power BI service
Power BI Desktop	Model builders Interactive report designers	Building Power BI dataset Building interactive reports
Power BI Report Builder	Paginated report designers	Building paginated reports
Excel	Analysts	Building Excel reports Analyzing Power BI datasets

Making the move

Moving away from Power Pivot won’t require any new services or infrastructure, and existing reports and models don’t need to be converted. They will continue to work and be supported for the foreseeable future. Microsoft has neither said not indicated that Power Pivot in Excel is going anywhere. However, by building your new datasets in Power BI Desktop, you will be better positioned moving forward.

If you do want to migrate some or all your existing Excel based Power Pivot datasets, it’s a simple matter of importing the Excel file into Power BI Desktop. This is completely different than connecting to an Excel file as a data source. From the File menu in Power BI Desktop, select Import, then select Power Query, Power Pivot, Power View. You will then select the Excel file that contains your dataset.

Power BI will then import all your Power Query queries, your Power Pivot dataset, and if you have any it will convert PowerView reports to the Power BI report types. The new report can then replace your existing Excel file. Once deployed to the Power BI service, other Excel files can connect to it if so desired.

Building your datasets with Power BI Desktop allows you to take advantage of a rich set of services, across a broad range of products, including Excel. Building them in Excel locks you into an Excel only scenario. If you already use Power BI, then there’s really no reason to continue to build Power Pivot datasets in Excel.

6 Comments

After adoption of SharePoint or Office 365, one of the first things an organization will look for is some understanding as to how the product is being adopted, and what its impact on resource allocation is. Historically, options for reporting on SharePoint have been limited at best.

The Web Analytics Service application was introduced with SharePoint 2010, and relied on a series of connected Excel workbooks and a fairly Byzantine series of staging and reporting databases. It worked so well that it was removed from the product in SharePoint 2013. The Usage logs contain a rich set of information, and they are rolled up into the Usage database, but accessing the data or persisting it beyond a short time period required a fair bit of work.

There were also third party analytical solutions, but most of these came with a hefty price tag, and they focused on page views, embedding code on a page. This approach works well enough for web pages, but it doesn’t capture everything, for example document access though the .NET API. They’re therefore not always well suited to collaborative environments.

SharePoint in Office 365 was initially devoid of analytics, but some basic reports have been creeping in in recent months. With the new administration portal going live, these reports moved from the relative obscurity of the compliance center to the brand new report center, and were augmented by some additional reports.

With the release of SharePoint 2016, and the announcements made at the Future of SharePoint Event on May 4 2016, we can see the additional areas where analytics are being introduced into the core product. At this point, it’s a good idea to step back and have a look at the Analytics landscape as it pertains to SharePoint and Office 365.

At the moment, the analytics offerings can be grouped into 4 major categories; tenant scoped, site scoped, document scoped, and Delve Analytics. Let’s have a look at each one in turn.

Tenant scoped

The tenant scoped reports are the aforementioned reports that are now available in the new Office 365 Reporting Center.

New usage reports for SharePoint OneDrive Yammer and Skype 1

There are a number of interesting reports in here that focus primarily on the tenant as a whole. How much OneDrive space users are using, Yammer message counts, Skype meetings, emails sent and received, etc. In addition, these reports can be interacted with to show four different time periods, 7, 30, 90, and 180 days. Year over year analysis is not available.

These reports will primarily interest administrators, and it therefore makes sense that they are only available in the administration center, where administrative permissions are required to access them.

Site Scoped

Site scoped analytics contain data that is of concern to site administrators. These users are more concerned with content usage than resource allocation. These analytics features were initially announced at the Future of SharePoint event on May 4 2016, and as of this writing, have not yet rolled out.

The initial rollout will focus on content consumption, visits to the site and document views

This is welcome data to beleaguered site administrators, and it will help to identify important content, and content that maybe could be pruned. While it will be initially rolling out to SharePoint Online, the good new is that on premises users will also be able to get this through the new Analytics service application.

In a similar model to the new hybrid search, the new Analytics service application called SharePoint Insights connects to Office 365 and delivers your on premises usage data to the service – essentially everything that is kept in the logging database. From there, the service can act on it to do interesting thing. One of those interesting things will be to deliver content based activity reports like the ones seen above.

There are a few things to take note of about site scoped analytics. They are scoped to the site, not the site collection. They do not roll up into a master report, so each site must be visited in turn (they live in the “site contents” section) to see the results. As far as I’m aware, the data is only persisted for a short time (I have only seen 7 days), so time based analysis is not possible.

Document scoped

Document scoped analytics have been in the service for some time now, and the new document library exposes them. I call them analytics, but they really are just the activity stream for a document or a folder. The do offer insight, so we’ll stick with the term.

From a “new style document library, you select the information icon on the right to open up the information pane. Part of that information is the activity stream of the document. In the example below I have selected a folder.

It’s a welcome addition, and it is what it is. There’s currently no way to aggregate the data or to pivot on it focused on anything either than the document/folder

Delve Analytics

Delve Analytics is a new offering from the Office team that focuses on the user. It analyzes a persons communications and schedule to provide insights into their work experience, with measures like time spent in meetings, time spent in email, work life balance, etc.

Take back your time with Delve Analytics 2

Delve analytics doesn’t really belong in a blog post about SharePoint because it doesn’t analyze any SharePoint or OneDrive data, so I’ll keep this section short. For the moment at least, it is restricted to Exchange email data as a source.

Delve Analytics requires an Office 365 E5 license or it can be purchased separately. Unlike the rest of the analytics options here, there is an extra cost associated with it.

Summary

The analytics options available in Office 365 and in SharePoint have improved drastically, but are still in their infancy. Each of the approaches are targeted at different audiences (IT Pro, site admins, content authors, individuals). This approach is bot good and bad. Tailoring an approach to an audience means that the specific audience will be satisfied, but the different approaches can become somewhat disjointed. It depends on what is necessary.

Analytics at the moment are also limited to specific time slices, if time can be sliced at all and to specific dimensions/measures. This is no problem if recent activity is the only thing of interest, but if more fine grained time slices or year-over-year analyses are needed, then the out of the box approaches will fall short.

Finally, most of the reports focus on activity, there is very little information provided about the SharePoint or Office 365 inventory.

The good news in all of this is not only that Microsoft has made analytics a priority, but that all of its features in this area use publicly available APIs. this means that there is plenty of room for third party vendors to step in to fill gaps and to provide complete analytics focused solutions. In that vein, I’m very proud to announce that my company, UnlimitedViz will soon be releasing a product, tyGraph for Office 365 to do exactly that.

2 Comments