Faster import & display with Data, Feeds, Views & Panels

Handling loads of data with nodes and fields in Drupal can be a painful experience: every field is put into a separate table which makes inserts and queries slow. In case you just want to import & display unstructured data without the flexibility and sugar of fields, this walkthrough is for you!  On a recent customer project, we were tasked with importing prices and other information related to products. While we are fine with handling 10k+ products in the database, we didn't want to create field tables for the price information to be attached to products. For every product, we have 10 maybe even more prices which would result in 100k+ prices at least. The prices shouldn't be involved in anything related to the product search, they should just appear as part of the product view itself. Also there is no commerce system involved at the current state of the project. Putting the prices into a separate field on the product node may sound like a good idea in the first place. Remember, when loading a list of of those products, all the prices will have to get loaded as well. We wanted those prices to be decoupled from the products, be stored in a lightweight way and only be loaded when necessary - on the single product view.

1) LIGHT-WEIGHT DATA STRUCTURES IN DRUPAL USING THE DATA MODULE

First, I thought implementing a custom entity or just data table would be the way to go. But then we considered giving the data module a try. The data module allows site builders to work on a much lower level than with Drupal fields: you can create database tables, specify their columns and define relationships. What it really makes appealing is that you can access the structured data using views, expose the custom data tables as custom entity types and use the Feeds module for importing that data, without any coding required. After installing the data module, you can manage your data tables under Structure > Data tables

We create a data table for the product prices and specify the schema with all the columns that should be included. Just like fields but without any fancy formatters on top of it:

This will create the desired database table for you. Having defined the data, we can use the Entity data module that comes with Data to expose the data table as a custom entity type. By doing so, you will get integrations like for example with Search API for free.

 

2) IMPORT USING FEEDS AND THE GENERIC ENTITY PROCESSOR

Luckily, the [Meta] Generic entity processor issue for the Feeds module has been committed after 3 years of work. As there hasn't been a release since the time of committing the patch (January 2014), this is only available from later dev versions of the Feeds module. But it's worth the hassle! We can now select from a multitude of different feeds processors based on all the different entity types in the system. After clearing caches, the data tables that we have previously exposed as entity types, do now show up:

The feeds configuration is performed as usual. In the following, we map all the fields from the clients CSV file to the previously defined columns of the data table:

We are now able to import large junks of data without pushing them through the powerful but slow Field API. A test import of ~30k items was performed within seconds. A nodes & Fields based import usually creates 200 items per minute.

3) DATA IS GOOD, DISPLAY IS BETTER

In the next step, we create a View based on the custom data table to display prices for products. We specify a number of contextual filters so that users will see prices a) the current product and restricted to b) the user's price source and c) currency.

Notice, that the Views display is a (Ctools / Views) Content pane, which has some advanced pane settings in the mid section of the views configuration. Most importantly, we want to specify the argument input: Usually we would use Context to map the views contextual filters to Ctools contexts that we provide through Panels.

Somehow, in this case, a specific field didn't work with the context system which automatically checks if all necessary context's are available and only allows you to use the Views pane under such circumstances. As you can see in the screenshot above, i have set all arguments to "Input on pane config" as a work around. Exactly these pane config inputs show up when we configure the Views pane in Panels. In this case, we have added the Product prices view as a pane on the panelized full node display of the Product node type (Drupal jargons ftw!).

Each pane config is populated with the appropriate keyword substitutions based on available contexts node and user of the panelized node.

4) THE END RESULT

Finally this is the site builded result of a product node including a prices table:   This concludes my how-to on the Data, Feeds, Views and Panels modules to attach a large data sets to nodes without putting them into fields. Once you know how the pieces fit together, it will take you less time than me writing this blog post to import and display large amounts of data in a less flexible, but more performant way!