To make it concrete, I’ll use MaxText’s make_hf_iterator as my reference code, and choose openwebtext-10k as our input dataset.

load raw dataset

First, let’s load the raw openwebtext-10k dataset. If streaming is on as shown below, data files will not be downloaded. Instead, it streams the data progressively while iterating on the dataset.

from datasets import load_dataset

dataset = load_dataset("stas/openwebtext-10k", split="train", streaming=True)

tokenize

At this stage, we need tokenize the raw dataset’s text field and trim the tokenized sequence up to predefined max_length. Practically, we’d first create a tokenizer either from a local file or from Huggingface via tokenizer_path. In the example below, we use t5-small tokenizer, which would be fetched from Huggingface directly.

# Sets some constants
add_bos, add_eos, max_length = True, True, 512
tokenizer_path = "t5-small"
data_column_name = "text"

# Creates a tokenizer
tokenizer = transformers.AutoTokenizer.from_pretrained(
    tokenizer_path,
    add_bos_token=add_bos,
    add_eos_token=add_eos,
    model_max_length=max_length,
    legacy=False,
)

Tokenization is then accomplished by running that tokenizer via dataset map function, _input_pipeline_utils.tokenization. This function applies the above tokenizer to the field data_column_name of each data example and truncates the tokens up to max_length.

from maxtext.MaxText.input_pipeline import _input_pipeline_utils

dataset = dataset.map(
    _input_pipeline_utils.tokenization,
    batched=True,
    fn_kwargs={"hf_tokenizer": tokenizer, "max_length": max_length - 1, "column_name": data_column_name},
)
# Post-tokenization: renaming the field where the tokens are.
dataset = dataset.select_columns(["input_ids"]).rename_column("input_ids", data_column_name)

transform: pack and shift

After tokenization, data examples become token sequences of various lengths. To increase training efficiency, we try to pack as many sequences as possible into the context window (max_length in the code). Here we use grain’s experimental packing API PackAndBatchOperation.

In multi-host setting, each host (i.e., process) gets an equal share of the global batch size (say 512), and this input pipeline code runs at host-level in parallel, thus we want host-level batch size when we batch, i.e., global_batch_size // jax.process_count().

import grain.python as grain
# Sets some constants.
global_batch_size = 512

# Adds packing transformation.
transformations = []
# HFNormalizeFeatures makes two copies of `text` field: one is called
# `inputs` and the other `targets`.
operations.append(_input_pipeline_utils.HFNormalizeFeatures(data_column_name))
transformations.append(
    grain.experimental.PackAndBatchOperation(
        # In multi-host setting, each host (i.e., process) gets an equal share 
        # of the global batch size.
        # And this input pipeline runs at host-level in parallel, thus we want 
        # host-level batch size here.
        batch_size=global_batch_size // jax.process_count(),
        length_struct={"inputs": max_length, "targets": max_length},
    )
)

# Post-packing: reformating tuple to flat dict style.
transformations.append(_input_pipeline_utils.ReformatPacking())

Finally we shift the inputs field by 1 token to the right, to make it ready for teacher-forcing training.

transformations.append(_input_pipeline_utils.ShiftData(axis=1))

sample

Now with all the transformations done, we need to tell each host how to sample from the transformed dataset. Most common settings include number of epochs (num_epochs), which shard of the dataset the current host should load (shard_options), whether to shuffle (shuffle), etc.

sampler = grain.IndexSampler(
    num_records=len(dataset),
    num_epochs=1,
    shard_options=grain.ShardOptions(
        shard_index=dataloading_host_index, shard_count=dataloading_host_count, drop_remainder=False
    ),
    shuffle=False,
    seed=0,
)

put together

We put everything together with grain.DataLoader API, which takes in the raw dataset, training-required transformations and sampler. The returned dataloader is ready to produce batches the downstream training loop needs (iter(dataloader)).

dataloader = grain.DataLoader(
    data_source=dataset,
    operations=transformations,
    sampler=sampler,
    worker_count=1,
    worker_buffer_size=1,
    read_options=grain.ReadOptions(num_threads=1, prefetch_buffer_size=128),
)

data_iter = iter(dataloader)
batch = next(data_iter)

Final words

Feel free to run and fork input_pipeline_data2iter.ipynb if you’d like to run a complete version of input pipeline. It’s worth noting that the returned batch sits in host CPU memory and so it’s necessary to further shard it across TPU devices before feeding the batch to pjitted train step. This could be done by MultiHostDataLoadIterator. If you’d like to know the details, this previous post could be of interest. If you’d like to run the input pipeline

This brief post focuses on the input pipeline in a multi-host setting. You can roughly think of multi-host as running programs over multiple GPU hosts or TPU hosts (each host has one or more workers called devices). The input pipeline performs at two stages:

• stage 1: it distributes the dataset across all hosts, i.e., each host gets a shard of the dataset.
• stage 2: at the local dataset shard, during iteration, it distributes the batch across the accelerator devices under the host (each TPUv5e host has 4 accelerator devices, while each GPU host typically has 1 accelerator device).

How do we achieve that in code? Since grain is becoming a popular choice (see comparison to TFDS and HuggingFace) in the Jax world, let’s dive into a grain implementation of such an input pipeline: preprocessing_pipeline. More concretely, let’s say we want to train a model with batch size 512 using 256 TPUv5e chips, i.e., 64 hosts.

• The first question is what batch size each shard should have, in our example, since global batch size is 512, each host is responsible for one 64th, so the local shard batch size is 8. (shard batch size):

batch_size=global_batch_size // jax.process_count()

• Then next question is which host gets which 64th shard of the dataset. This is specified by grain.IndexSampler and its shard_options argument (code-link):

shard_options=grain.ShardOptions(
    shard_index=dataloading_host_index, shard_count=dataloading_host_count
)

For the first host, shard_index is 0, and shard_count is 64.

• Now for stage 2, each host will be distributing local batches of size 8 across 4 devices. MaxText has this iterator class MultiHostDataLoadIterator, which takes in a dataloader and turns it into an iterator (local_iterator), and its __next__ method will do the batch distribution.

• The actual heavy lifting (distributing) is done by _form_global_array. First, it splits the batch array into N pieces (N is the number of local devices) and then put each piece to assigned device in order. Finally, it informs jax that those local arrays form a global array (this becomes more relevant when we talk about pjit in future posts.)

Some final thought, it’s not always the case to distribute the host batch across all local devices; it depends on data_sharding configuration. We’ll probably dive deeper into this later, for now, distributing all the way to each local device is a good starting point.

if you have comments or suggestions or spotted an error, please let me know via email: kehanghan at gmail dot com.

One might think it’s not really necessary to voice control a desk to go up and down since the buttons on the control panel already do these very easily. Two reasons for me to carry out this version:

• introducing a new communication mechanism (i.e., natural language) to my desk is technically very interesting; as you will see, it involves designing a whole series of modules that work together such as wake word detection, speech recognition, intent classification and execution.

• adding voice-control actually makes it possible to have more sophisticated desk behaviors. For instance, asking the desk to rest for 10 minutes and then go back to work again, or start a working schedule at 10 am on Monday.

Here is how it looks: video demo (code).

Voice-control workflow

Just like Google Home Assistant, for voice control to work, we have to build a system of four components and coordinate them to work together.

• wake word detection module listens to the audio stream constantly and triggers speech recognition module when it picks up certain words (e.g., Hey Goolge).

• speech recognition module converts audio waveform into text representation.

• intent detection module takes in a text sentence and figures out the intent and associated parameters (e.g., rest for 10 mins can be classified as the intent of going down with argument 10 mins).

• based on the intent, corresponding APIs can be used to acutally execute the actions.

I decided to use Bob as my wake word (so that becomes the name of my desk). Not to complicate things in the first try, I built the logic flow via the following code:

# Check if the wake word is present in the speech
if is_wake_word(text, wake_word):
	# Wake word has been detected, do something...
	print("Wake word detected:", text)
	intent, kwargs = intent_detection.detect_intent(text)
	print("Intent detected:", intent, kwargs)
	execute(intent, **kwargs)

Wake word detection

In this prototype, my detector is made extremely simple using library Uberi/speech_recognition: check if Bob appears in the text sentence recognized by sr.Recognizer(). For advanced use cases, one may find this library useful Picovoice/porcupine.

Speech recognition

Uberi/speech_recognition provides multiple APIs including CMU Sphinx, Google Speech Recognition, Microsoft Azure Speech etc. I used Google Speech Recognition in this post and found that very easy to hook up with. The library even provides a generic API key so if you don’t use it too heavily you can call the API out of the box. If one feels like not to rely on those APIs from big tech, it’s not a bad idea to build a speech recognizer from scratch via machine learning. Here is an example speech recognition by Michael Phi.

Intent detection

Intent detection is also a good place to utilize machine learning. Here again not to complicate things, I built a keyword-based solution to classify intents.

# Define the keywords that indicate each intent
stand_up_keywords = ["stand", "up", "rest"]
sit_down_keywords = ["sit", "down", "work"]
report_height_keywords = ["height"]

# Check if any of the keywords for each intent appear in the text
if any(word in text for word in stand_up_keywords):
    intent = "rest"
    # Extract the time duration information from the command text
    kwargs = {"timeout": extract_duration(text)}
...
...

Acknowledgements

I’d like to thank a couple of projects here:

• Uberi/speech_recognition provides the backbone of my voice-control system.
• This is the first time I used a machine/AI for pair-programming, yes you guessed right chatGPT, which certainly made the whole development process delightful.

I tend to forget to stand up and rest during work, so the first version presented here is to give a schedule to my desk. Here is how it looks: video demo (code).

High level connection

A standing desk usually uses a control panel/pad to send commands (up, down, etc.) to the desk’s motor controller via a RJ45 cable (Ethernet cable).

In my case, I’ll replace the control panel with a Raspberry Pi (that way, we can implement some intelligence logic on the Pi side, which becomes the brain of our desk). How to connect? We can still use a RJ45 cable with an additional component: RJ45 breakout board. The breakout board flattens the 8 pins of RJ45 in a way that’s super convenient to wire with Pi’s GPIO pins.

Low level connection

Importantly we need to connect 4 pins of the RJ45 with 4 GPIO pins of the Pi. The exact mapping depends on the desk model. In my case, it’s a Flexispot EC3 model which uses the HS11A-1 control panel. The information flows like this: command (in bytes format) gets emitted from the Pi’s pin 8 (i.e., TX), passes through RJ45 pin 6 and arrives in motor controller’s RX. data like desk height (in bytes format) get emitted from the motor controller’s TX, passes through RJ45 pin 5 and arrives in Pi’s pin 10 (i.e., RX).

Note other models using different control panels may have RJ45 pins arranged differently. (let me know if you’d like some details on how to debug pins for other models)

Command encoding

When commands are sent to controller, they have to be a stream of bytes. Thanks to nv1t/standing-desk-interceptor and iMicknl/LoctekMotion_IoT, below is a mapping between the major commands and their bytes format. For instance, the up command is encoded as \x9b\x06\x02\x01\x00\xfc\xa0\x9d.

Note the preset commands make the desk go to the preset heights.

Height data decoding

In reverse direction, height data gets sent from controller to the Pi’s RX pin. The data is also encoded in bytes. Below is an example packet.

The encoded height has three byte characters, each representing a digit in the height. For instance, \x06 → 1 (read futher on why such mapping is made). I’m listing the mapping of 10 digits to the bytes down below.

It’s worth noting that a digit with a decimal point is encoded differently than a pure digit. So here’s another mapping just for that.

Thus the above example of encoded height \x5b\xff\x06 gets decoded to 28.1 inch.

Scheduling

Based on my personal habit, I created this scheduling script to split each hour for my standing desk: 50 min work + 10 min rest. Each day the desk runs 8 rounds of work and rest. A crontab job is scheduled to run this script on desired days.

Acknowledgements

I’d like to thank several projects here:

• LoctekMotion_IoT has a very good README, summarizes findings from other projects and covers a lot of basics.
• I learned how to decode the height data from LoctekReverseengineering.
• alselectro has a detailed and educational post on how a digit gets rendered via 7-segment display, basically why 1 is represented by \x06.

• popular traditional models

• deep learning models

• python libraries

Over the history of NER, there’s been three major approaches: grammar-based, dictionary-based and machine-learning-based. Grammar-based approach produces a set of empirical rules hand-crafted by experienced computational linguists, usually takes months of work. Dictionary-based approach basically organizes all the known entities into a lookup table, which can be used to detect whether a candidate belongs to a defined category or not. By design it doesn’t work well with newly invented entities. Machine-learning-based approach typically needs annotated data, but doesn’t necessarily rely on domain experts to come up with rules or fail on unseen entities.

This post focuses only on machine-learning based models.

1. Popular traditional models

The traditional models we’ll discuss here are MEMM, CRF. They are very popularly used before deep learning models entered the scene.

1.1 MEMM

We’ve covered the details of MEMM in the previous post. The key idea of the MEME approach is to model the conditional probability of tag sequenece for a given sentence with Markov assumption:

We then model \(p(y_i | y_{i-1}, x_1...x_n)\) using local environment:

In inference, we use Viterbi algorithm to get best-fitting tag sequence for a given sentence. Details can be found in 2.2.1 section of the previous post.

In training, we use maximum likelihood estimation to get optimal \(\underline{\theta}\) that

where \(\underline{x}^j, \underline{y}^j\) are the \(j^{th}\) sentence and corresponding tag sequence (the whole training dataset has \(N\) examples).

1.2 CRF

Instead of \(p(y_i | y_{i-1}, \underline{x})\) , Conditional Random Field (CRF) approach chooses to directly model \(p(\underline{y} | \underline{x})\):

The main challenge in direct modeling is that the denominator is sum of \(K^n\) terms where \(K\) is the number of tag label types and \(n\) is the length of sentence to tag. This is a much larger number than that in MEMM - \(p(y_i | y_{i-1}, x_1...x_n)\) has just \(K\) terms in the denominator.

1.2.1 Inference

During inference, we are only interested in the \(\underline{y}^{*}\) that gives the highest probability rather than the highest probability itself:

If using brutal force, we have to evaluate \(\exp({\underline{\Theta} \cdot \underline{F}(\underline{x}, \underline{y})})\) for \(K^n\) times.

Fortunately, if we add a little structure into \(\underline{F}(\underline{x}, \underline{y})\), which I’m going to talk about next, we can bring the exponential complexity - \(O(K^n)\) down to linear complexity - \(O(K^2n)\).

The structure added in CRF is:

To maximize \(\underline{\Theta} \cdot \underline{F}(\underline{x}, \underline{y})\), we define a partial score as we did in 2.2.1 section of the previous post:

If we can maximize any partial score (which turns out not that difficult), then the score we want to acutally maximize, \(\underline{\Theta} \cdot \underline{F}(\underline{x}, \underline{y})\), is just a special case of \(s_{partial, k}\) when \(k=n\).

So how to maximize any partial score? Let’s start with \(k=1\), namely \(s_{partial, 1} (y_1)= \underline{\Theta} \cdot \underline{f}(\underline{x}, y_1)\).

This is easy because it’s just a single-variable optimization and \(y_1\) can only have K choices. We also store all the evaluated \(s_{partial, 1} (y_1)\).

How about \(k=2\), namely maximize \(s_{partial, 2}(y_1, y_2) = s_{partial, 1}(y_1) + \underline{\Theta} \cdot \underline{f}(\underline{x}, y_1, y_2)\)?

We can first fix \(y_2\) and optimize over \(y_1\) dimension. Remember we’ve known \(s_{partial, 1}(y_1)\) evaluated from the previous question. So it takes \(K\) computations to find the optimal \(y_1\) for each \(y_2\) - \(s_{partial, 2}(y_1^*, y_2)\). Then pick the \(y_2^*\) which has maximum \(s_{partial, 2}(y_1^*, y_2)\). In total, we need perform \(K^2\) evaluations. We also store all the \(s_{partial, 2}(y_1^*, y_2)\) for future use.

How about \(k=3\), namely maximize \(s_{partial, 3}(y_1, y_2, y_3) = s_{partial, 2}(y_1, y_2) + \underline{\Theta} \cdot \underline{f}(\underline{x}, y_2, y_3)\)?

Similar to the previous question, we try to estimate \(s_{partial, 3}(y_1^*, y_2^*, y_3)\) for each \(y_3\) using \(s_{partial, 3}(y_1^*, y_2^*, y_3) = \max_{y_2}(s_{partial, 2}(y_1^*, y_2) + \underline{\Theta} \cdot \underline{f}(\underline{x}, y_2, y_3))\). We also carry out \(K\) evaluation per \(y_3\), thus totally \(K^2\) evaluations for all possible \(y_3\). We store \(s_{partial, 3}(y_1^*, y_2^*, y_3)\) for future use (e.g., when \(k=4\)).

By doing this all the way to \(k=n\), we can get \(\max_{\underline{y}}{\underline{\Theta} \cdot \underline{F}(\underline{x}, \underline{y})}\) with roughly \(K^2n\) evaluations.

1.2.2 Training

Similar to MEMM, we can also use maximum likelihood estimation to get optimal \(\underline{\Theta}\) that

where \(\underline{x}^j, \underline{y}^j\) are the \(j^{th}\) sentence and corresponding tag sequence (the whole training dataset has \(N\) examples). More details on training algorithm can be found in Page 10 of Michael Collins’s CRF note.

2. Deep learning models

The deep learning models we’ll discuss here are LSTM, BiLSTM-CRF, Bert.

2.1 LSTM

2.1.1 Architecture

In the setting of LSTM, each token \(x_i\) is fed to a LSTM unit, which outputs a \(o_i\). \(o_i\) models log probabilities of all possible tags at i-th position, so it has dimension of \(K\).

2.1.2 Inference

The inference in LSTM is very simple: \(y_i\) = the tag with highest log probability at i-th position.

which indicates the prediction of i-th position only utilizes the sentence information up to i-th token - only the left side of the sentence is used for tag prediction at i-th position. BiLSTM is designed to provide context information from both sides, which will be seen in next section.

2.1.3 Training

Like all the other neural network training, LSTM training uses Stochastic Gradient Descent algorithm. Loss function adopts negative log likelihood. For a data point \((\underline{x^j}, \underline{y^j})\), we have its loss calculated as:

where \(n_j\) is the length of the sentence \(x^j\), \(o_i^j\) is the LSTM output at i-th position and \(y_i^j\) is the ground truth tag at i-th position.

Total loss is the mean of all the individual losses.

where \(N\) is the total number of training examples.

2.2 BiLSTM

BiLSTM stands for bi-directional LSTM, which provides sequence information from both directions. Because of that, BiLSTM is more powerful than LSTM. Except the bi-directional component, the meaning of network output, inference, and training loss are same as LSTM.

2.3 BiLSTM-CRF

BiLSTM captures contextual information around i-th position. But at each position, BiLSTM predicts tags basically in an independent fashion. There’s cases where some adjacent positions are predicted with tags which do not usually appear together in reality. For example, I-PER tag should not follow B-ORG. To account for this kind of interactions between adjacent tags, Conditional Random Field (CRF) is introduced to BiLSTM.

2.3.1 Architecture

where \(o_i\) models emission scores of all possible tags at i-th position and \(y_i^*\) is the best tag for i-th position which collectively achieves highest sequence score.

CRF layer also learns a transition matrix \(A\) which stores transition scores between any possible pair of tag types.

2.3.1 Inference

Same as the inference in CRF section, given a trained network and sentence \(\underline{x}\), any sequence \(\underline{s}\) will have a score.

The score is a sum of contributions from token level. i-th position has contribution of \(\phi(\underline{x}, s_{i-1}, s_i) = o_i[s_i] + A[s_{i-1}][s_i]\), where the first term is emission score and second term is transition score.

To find the tag sequence \(\underline{y}^*\) achieving highest score, we need to use dynamic programming.

Define sub problem \(DP(k,t)\) to be the max score accumulated from 1st position to \(k\)-th position with the \(k\)-th position tag being \(t\), detailed as follows:

The recursion would be:

The original problem is then

We can always use parent pointers to retrieve the corresponding best sequence \(\underline{y}^*\).

2.3.2 Training

Loss function for BiLSTM-CRF also adopts negative log likelihood. For a data point \((\underline{x^j}, \underline{y^j})\), we have its loss calculated as:

where the first term is easy to calculate via a forward pass of the network and the second term needs more care. Let’s define that term (without log) as \(Z\), which is exponential sum of scores of all the possible sequences \(\underline{s}\) of length \(n\).

To calculate \(Z\), we need to use dynamic programming again. This time the sub-problem \(DP(k,t)\) is the exponential sum of scores of all possible sequences of length \(k\) with last tag \(s_k = t\):

The recursion would be:

The original problem is then

Via this way, individual loss \(L_j\) is calculated and then batch loss by averaging the individual losses in the batch.

2.4 Bert

Recent research on BERT provides an option for NER modeling. Despite of the complexity of the BERT model architecture, in the context of NER it can be regarded as an advanced version of our BiLSTM model - replacing the LSTM with multiple Transformer Encoder layers.

Thus, \(o_i\) still models log probabilities of all possible tags at i-th position.

Inference, and training loss are same as LSTM section.

3. Python libraries

There’s several machine learning based NER repositories in GitHub. I picked some of them here with some comments.

• KEHANG/ner: for English texts, based on PyTorch, has LSTM, BiLSTM, BiLSTM+CRF and Bert models, has released conda package

• shiyybua/NER: for Chinese texts, based on Tensorflow, only BiLSTM+CRF model, no packages released

• Franck-Dernoncourt/NeuroNER: for English texts, based on Tensorflow, has LSTM model, no package released

• what is NER

• how to formulate it

• what are the traditional and start-of-the-art models

• what are the off-the-shelf options

• how to build a customized NER model

So this post will try to provide a complete set of explanation on these questions.

1. What is NER

Simply put, Named Entity Recognition is a technology that identifies certain entities from a sentence/paragraph/document. Like the sentence below, NER tries to tag each word with a label; Steve Jobs and Steve Wozniak to be PER (persion entity), Apple to be ORG (organization entity) and the rest O (not an entity).

This is useful in lots of cases. For instance, Apple Mail identifies TIME entity in emails and makes pulling events from email to calendar much easier than before; Google Search is able to find Headquarters entity in a relevent document to answer query questions.

So how does it actually work?

2. Mathematical Formulation

Let’s step back and see what NER is doing essentially. For a given sentence \(x_1 ... x_n\), NER decides to tag each word \(x_i\) with an entity label \(y_i\).

One obvious challenge here is one word should be tagged differently depending on its context. E.g., Apple in the example above is an organization entity, but in other contexts might be a fruit entity.

So how can we tag wisely? Basically we need two things.

• a score function \(s(y_1...y_n, x_1...x_n)\), which splits out a score measuring how fit a tagging sequence \(y_1 ... y_n\) is to a given sentence \(x_1 ... x_n\). A well-designed score function should assign a higher score to a tagging sequence that fits better.

• a solver, which is able to pick the highest-scoring tagging sequence among overwhelmingly large number of possible candidates. Just take a sentence with \(7\) words as an example, we are talking about \(10^7\) tagging candidates if there’s 10 unique entities to choose from. A good solver should be able to efficiently get to the best tagging sequence.

2.1 Score Function

Researchers in the field like to use probability model to build score function: a better-fitting sequence can be given a higher probability. Often times we choose conditional probability (one can use joint probablity as well), like below

If we make a simplication \(p(y_i |y_1...y_{i-1}, x_1...x_n) \approx p(y_i | y_{i-1}, x_1...x_n)\), then

Now the question becomes how do we model

2.1.1 Model \(p(y_i | y_{i-1}, x_1...x_n)\)

Following the above example, we are basically asking how likely Jobs is PER / ORG / MISC / O, if Steve is PER?

A natural thought would be to create a second score function (let’s call it local score function since it looks only at \(y_i, y_{i-1}\)) with well-defined local features \(\vec{f}(y_{i-1}, y_i, x_1...x_n)\):

and define probability based on the local score function:

\(\vec{f} \in R^m\), is an m-dimension vector, meaning there’s m predefined feature functions \(f_1...f_m\). Some feature examples could be:

At this point, once we have defined \(\vec{f}\) and picked the weights \(\theta\) for the features, we can readily calculate \(p(y_i | y_{i-1}, x_1...x_n)\) and tagging score from

2.1.2 Three remaining questions

Let’s summarize here. With this framework set up, there’s only 3 questions remaining:

• how to design feature functions \(\vec{f}(y_{i-1}, y_i, x_1...x_n)\)?

Traditionally it’s kinda art and requires expert knowledge in the problem domain. But recent deep learning approaches such as LSTM, BiLSTM aim to utilize large neural networks and automatically figure out suitable featurization from data.

• how to estimate weights \(\vec{\theta}\)?

\(\vec{\theta}\) can be obtained by training on data. One intuitive and popularly used method is Maximum Likelihood Estimation.

• assuming we already know \(\vec{f}, \vec{\theta}\), how to get the best fitting tag sequence \(y_1^{*}...y_n^{*}\) for a sentence \(x_1...x_n\)?

This will be the central topic in the next section - Solver section.

2.2 Solver

Now let’s assume we’ve determined \(\vec{f}, \vec{\theta}\) from expert knowledge and/or data, so our score function \(s(y_1...y_n, x_1...x_n)\) is finalized.

The job of the solver here is use the score function to efficiently get the best fitting tag sequence \(y_1^{*}...y_n^{*}\) for a given sentence \(x_1...x_n\). It’s an optimization problem:

We can definitely solve it by brutal force - it’s just that there’s \(5^7\) possible \(y_1...y_n\) sequences in the example below, meaning we have to evaluate \(5^7\) times of our score function before getting the optimal sequence.

The computation scales badly with the length of sentence.

The computation complexity is exponential - \(O(5^N)\) where \(N\) is the length of sentence.

It turns out the score function exhibits a special mathematic structure that enables us to solve the optimization problem in linear time \(O(N)\). Let’s talk about that.

2.2.1 Viterbi Algorithm

We notice our score function is a product of successive conditional probabilities \(p(y_i |y_{i-1}, x_1...x_n)\) as below:

You’ll see this is a very good property that allows us to optimize step by step via dynamic programming (in this case we call Viterbi Algorithm). So, bear with me for a bit…

Because of this nice and clean form, we can easily define a partial score function \(s_{partial}\) - score of the first \(k\) tags \(y_1...y_k\) for the same given sentence \(x_1...x_n\).

When \(k=n\), the partial score function becomes the original score function.

We can also easily find the recursive form of the partial score function:

Now, we only need answer three questions to explain the algorithm.

• How would you find highest partial score where \(k=1\), namely \(s_{partial, 1}(y_1^{*}, x_1...x_n)\)?

This is easy because it’s just a single-variable optimization and \(y_1\) can only have 5 choices.

We can just evaluate \(s_{partial, 1}(y_1, x_1...x_n)\) for 5 times and pick the highest one. To help answer the following questions, we’ll save the 5 evaluated numbers. Here we carried out \(5\) evalutions to get \(s_{partial, 1}(y_1^{*}, x_1...x_n)\).

• How would you find highest partial score where \(k=2\), namely \(s_{partial, 2}(y_1^{*}, y_2^{*}, x_1...x_n)\)?

It’s just bi-variable optimization. We can first keep \(y_2\) fixed and optimize over \(y_1\) dimension.

We can calculate the best \(s_{partial, 2}(y_1^{*},y_2=\text{PER}, x_1...x_n)\) by evaluating the following 5 products and pick the best one.

1. \[p(y_2=\text{PER} |y_1=\text{PER}, x_1...x_n) \cdot s_{partial, 1}(y_1=\text{PER}, x_1...x_n)\]
2. \[p(y_2=\text{PER} |y_1=\text{ORG}, x_1...x_n) \cdot s_{partial, 1}(y_1=\text{ORG}, x_1...x_n)\]
3. \[p(y_2=\text{PER} |y_1=\text{LOC}, x_1...x_n) \cdot s_{partial, 1}(y_1=\text{LOC}, x_1...x_n)\]
4. \[p(y_2=\text{PER} |y_1=\text{MISC}, x_1...x_n) \cdot s_{partial, 1}(y_1=\text{MISC}, x_1...x_n)\]
5. \[p(y_2=\text{PER} |y_1=\text{O}, x_1...x_n) \cdot s_{partial, 1}(y_1=\text{O}, x_1...x_n)\]

Note we can reuse \(s_{partial, 1}(y_1, x_1...x_n), y_1 \in \{\text{PER, ORG, LOC, MISC, O}\}\) saved from previous question.

In this manner, we have all of \(s_{partial, 2}(y_1^{*},y_2, x_1...x_n), y_2 \in \{\text{PER, ORG, LOC, MISC, O}\}\) ready and the highest of the five would be the answer. Here we carried out \(5*5=25\) evalutions to get \(s_{partial, 2}(y_1^{*}, y_2^{*}, x_1...x_n)\).

• How about \(k=3\) and beyond?

At this moment, you might have noticed that we could get \(s_{partial, 3}(y_1^{*}, y_2^{*}, y_3^{*}, x_1...x_n)\) by evaluating \(s_{partial, 3}(y_1^{*},y_2^{*}, y_3, x_1...x_n), y_3 \in \{\text{PER, ORG, LOC, MISC, O}\}\) and picking the highest of the five.

Each \(s_{partial, 3}(y_1^{*},y_2^{*}, y_3=u, x_1...x_n)\) can be easily calculated via

Note we can reuse \(s_{partial, 2}(y_1^{*}, y_2, x_1...x_n), y_2 \in \{\text{PER, ORG, LOC, MISC, O}\}\) saved from previous question.

Similarly, it takes another 25 evalutions. Basically we can keep rolling to all the way \(k=n\) with each step forward we carry out another 25 evaluations. So the total compuation complexity is around \(25N\), namely \(O(N)\).

This post turns out to be longer than I thought, I was only able to answer the first questions raised in the beginning. And this seemingly natural approach is actually the so-called maximum entropy Markov model (MEMM) approach. There’s many more other ways to solve NER problem, which I’ll talk about in next posts together with the answers to the remaining three questions. See you soon.

In that process, one of the most challenging problems is to predict thermodynamic properties (e.g., enthalpy, entropy, heat capacities) for any molecules in the chemical space, of which most we may have never seen before.

In machine learning language, molecular prediction research is essentially answer how to smartly map a molecule, regarded as a node-edge graph, to a feature vector and then design appropriate regression model to get to molecular property.

This post reviews several traditional approaches (used as base-line models) and propose a new solution using Molecular Convolutional Neural Networks, which boosts prediction performance remarkably.

0. Datasets

The datasets we are using are crafted originally from a large quantum-mechanics calculation dataset named ScientificData-134k (thereafter sdata134k) by Ramakrishnan. Since the most difficult-to-predict molecules always involves cyclic structures, this post gathers all cyclic molecules from sdata134k and categorizes them into into hydrocarbon cyclics and oxygen-contained cyclics.

Dataset 1: sdata134k-hydrocarbon-cyclic examples

Dataset 2: sdata134k-oxygenate-cyclic examples

1. Base-line models

The traditional approaches that will be covered and used as base-line models includes

• Linear model: Group additivity method

• Molecular fingerprint with neural netoworks

Base-line model 1: Group addtivity method

Historically, Benson Group Additivity Method is polularly used because of its simplicity. It breaks down a molecule into sub-structural pieces and sums up their contributions to overall thermodynamic properties, which essentially is a linear model with bag of sub-structures.

For example, the molecule 2-methylnonane consists of three types of sub-structures includes:

• 1 tertiary carbon atom
• 6 secondary carbon atoms
• 3 primary carbon atoms

This simple idea is effective for linear-shaped molecules, but for the cyclic molecules collected in the above two datasets, Benson Group Additivity approach (already implemented in RMG thermo estimator) performs badly:

• Enthalpy energy prediction test error = 60 kcal/mol (RMSE).

The error largely comes from its underlying design/assumption:

• the effect of relative positions of sub-structures is negiligle (independent contributions from sub-structures)
• sub-structures are very small, usually atom based

The assumptions become less valid when it comes to cyclic compounds since they are relatively large molecules with strong interations between sub-structures.

Base-line model 2: Morgan fingerprint with neural net

Morgan fingerprint, also known as ECFP (Extended-Connectivity Fingerprint) tries to see bigger chemical environments within a moleucle. It defines fragments with user-specified radius (e.g., 2-atom distance radius). More details can be found in RDKit presentation.

In this base-line model, we use Morgan fingerprint as our molecular feature vector. We also use one-hidden-layer neural networks to account for non-linear interations between fragments.

The architecture is shown below:

This approach of combining Morgan fingerprint and non-linear regression model improves prediction performance:

• Enthalpy energy prediction test error = 19 kcal/mol (RMSE).

However, 19 kcal/mol is still not satisfying in chemistry discovery; to our experience, enthalpy energy uncertainty of larger than 5 kcal/mol could easily mis-guide discovery direction.

2. Molecular convolutional neural network

Besides accuracy issue, for fixed molecular fingerprint, people face the practical challenge that feature engineering requires chemical expertise and manual tweaking, naturally slowing down the process of learning from data.

To solve that, one idea by Aspuru-Guzik is to make fingerprint also learnable from data. Thus, this project is actually to build an accurate enthalpy estimator that learns featurization and regression altogether from data.

More specifically, the learnable featurization module uses graph convolutional neural networks (see architecture below). The learned feature vector willthen be fed into a fully connected neural network with one hidden layer, which serves the regression model.

Architecture

Each input molecule to molecular convolutional neural network (thereafter MCNN) is represented by three components: atom fingerprint matrix (denoted by A), bond fingerprint tensor (denoted by B), connectivity matrix (denoted by C).

For dimensionality, \(A\) is \(n_a × f_a\), \(B\) is \(n_a × n_a × f_b\), and \(C\) is \(n_a × n_a\), where \(n_a\) is the number of atoms, \(f_a\) the size of atom fingerprint and \(f_b\) the size of bond fingerprint. The output of MCNN is molecular fingerprint vector with size of \(f_m\).

Atom fingerprint matrix \(A\)

For a given molecule, \(A\) stores the information at atom level; each atom has a fingerprint vector therefore \(A\) has size of \(n_a × f_a\). Atom fingerprint includes basic atomic information such as atomic charge, number of attached hydrogens, etc.

Bond fingerprint matrix \(B\)

For a given molecule, \(B\) stores the information at bond level; each bond has a fingerprint vector therefore \(B\) has size of na × na × fb. Bond fingerprint includes basic information such as bond order, whether the bond is in ring, etc.

Connectivity matrix \(C\)

For a given molecule, each possible pair of atoms has an entry in \(C\); 1 indicates there’s a bond between the pair, vice versa, therefore \(C\) has size of \(n_a × n_a\).

Molecular convolution

We could add each atom fingerprint to its local neighbors by some weights if neccessary, called molecular convolution, using \(A\) and \(C\), and get convoluted atom fingerprint matrix \(A^{r=1}\). r = 1 indicates the output fingerprint contains local information with radius of 1 atom distance. Increasing radius by consecutive convolution gives \(A^{r=m}\) which captures larger local neighborhood information.

Performance

The best model selected from 5-fold cross-validation gives remarkable preformance boost:

• Enthalpy energy prediction test error = 3 kcal/mol (RMSE).

Interpretation

Let’s see if the learned fingerprint makes sense.

Check 1: similar molecules with similar fingerprint

Three similar cyclic hydrocarbon (first three in the figure) are selected, whose fingerprints share major high peaks. A linear molecule with same number of carbons appears relatively different, as expected. So, pass!

Check 2: king-queen =? man-woman

In word2vec project, researchers tends to use subtraction to interpret embedding vector (word equivalent of fingerprint). One of the most famous examples is using word2vec, vector(king)-vector(queen) is very similar to vector(man)-vector(woman), as semantics requires.

Similarly, four molecules are selected:

Again, the king molecule and queen molecule differ by a 4-member ring, same as for the difference between man molecule and woman molecule. The two fingerpring differences are quite similar, sharing most major high peaks. Another pass!

Check 3: nearest neighbor in fingerprint space

Although there’s no absolute definition of molecular similarity, human can generally tell if two molecular structures are alike. This check is to see if the machine generates sensible similar molecules in the training set given any molecules.

The given molecule example in the figure consists of a 4-member ring and 6-member ring; since 6-member and 5-member are quite similar (both have similar ring strain contributions), we see most returned neighbours have a 4-member ring and either one 5-member or one 6-member ring. Looking good!

Another aspect is unsaturated bonds in the given molecule: one double bond next to the fused bridge and one far-away triple bond. The second neighbour from left is exactly matching this aspect and all the remaining neighbours also have two similar unsaturated parts in structure. Pass!

At this point, we’ve kind of got a new model/estimator that appears to have greatly better accuracy and generate sensible fingerprints (although we still don’t exactly know what each entry in the fingerprint vector stands for).

3. Effectiveness of learning new data

Besides accuracy and interpretability, it’s also crucial to evaluate how effectively the new machine architecture learns from new data.

It is because the range of applications RMG explores keeps extending; along the past 10 years, RMG started from modeling nature gas combustion (usually small simple molecules) to recently being capable of handling full chemistry of hydrocarbon up to 20 carbons (obviously a lot new molecular structures come into our scope). On the other hand, it’s extending its capability towards oxygenates, sulfurides and nitrogen compounds as well.

So, here we prepared two extra small datasets, whose molecules are distinct from those in the above training datasets.

• Large tricyclic: larger rings that those in previous training datasets but mostly have saturated bonds as previous training examples do

• Unsaturated cyclics: cyclics that have multiple unsaturated bonds in same ring, which is very distinct from preivous training examples.

As expected, the unsaturated cyclic dataset starts with higher test error than large tricyclic one, which is because its examples are much more different from previous training examples than the other.

Good news is both datasets have test error go down as feeding more new relavent data points. After 50 new data points, the test error enters 10 kcal/mol region.

4. Future work

The new proposed estimator appears much effective in predicting molecular themodynamic properties by mapping graph structures through fingerprints to eventually properties. To expand its predicting power to new categories of molecules, we’d make sure it can learn as long as new data is available. But data is a must to make this estimator evolve. To accomplish that, I’ll talk about data insfrastructure construction in next posts to build a data-machine pipeline.

This report will be available online after due date at: http://kehang.github.io/

for people (including graders) to review. After grading, the post will be modified to be a standalone report with more background.

Package specifications for this project:

• OS: Mac

• python: 3.6.0

• keras: 2.0.2

• backend of keras: theano

• theano: 0.9.0

1. Multinomial/Softmax Regression

Part 4: report base-line test error

when temperature parameter = 1, the test error is 0.1005, implying the linear softmax regression model is able to recognize MNIST digits with around 90%.

Part 5: explain temperate parameter effects

Increasing temperature parameter would decrease the probability of a sample \(x^{(i)}\) being assigned a label that has a large \(\theta\), and increase for labels with small \(\theta\). The mathematic explanation is following:

The first term is the bracket is weighted average of \(\theta x\), so if \(\theta_j x\) is large, the value of the brackect will be negative, leading to negative \(\frac{\partial log(P_j)}{\partial \tau}\).

For small \(\theta_j\), \(\theta_j x\) is also small, we have positive \(\frac{\partial log(P_j)}{\partial \tau}\).

Part 6: temperate parameter effect on test error

During experimentation, we found the test error increases with temperature parameter as follows:

when temperature parameter = 0.5, the test error is 0.084

when temperature parameter = 1.0, the test error is 0.1005

when temperature parameter = 2.0, the test error is 0.1261

Since from Part 5 we know increasing temperature parameter makes probability of large-\(\theta\) label decrease and that of small-\(\theta\) label increase, the probability distribution becomes more uniform as temperature parameter icreases.

Part 7: classify digits by their mod 3 values

Note: in this part we use temperature parameter = 1.0.

The error rate of the new labels (mod 3) testErrorMod3 = 0.0768.

The error rate of the original labels testError = 0.1005.

The classification error decreases because those examples being classified correctly with original labels will continue being classified correctly with new labels, but those being classified wrongly originally will have a chance to be classified correctly with new labels.

Part 8: re-train and classify digits by their mod 3 values

After re-training using the new labels (mod 3), testErrorMod3 = 0.1872.

Part 9: explain test error difference between Part 7 and 8

Compared with testErrorMod3 = 0.0768 in Part 7, re-training makes the test accuracy worse. It is because now in training step, the algorithm has less information (very different digits may have same mod 3 labels) with new labels compared with training using 0-9 digit labels.

2. Classification using manually-crafted features

This section we explore one dimensionality reduction approach: PCA and one dimensionality increase approach: polynomial (specifically cubic version) feature mapping, to see their effects on test error rates.

Part 3: report PCA test error

When we use dimensionality reduction by applying PCA with 18 principal components, the test error = 0.1483. This error rate is very similar to the original d=784 case. It is because PCA ensures these 18 feature values capture the maximal amount of variation in the original 784-denmensional data.

Part 4: first 2 principal components visualization

Below is the visualization of the first 2 pricipal components of 100 training data points.

Part 5: image reconstruction from PCA-representations

Below are the reconstructions of the first two MNIST images fron their 18-dimensional PCA-representations alongside the originals.

MNIST 1st Image (left: reconstructed, right: original)

MNIST 2nd Image (left: reconstructed, right: original)

Part 6: explicit cubic feature mapping for 2-dimensional case

For \(x = [x_1, x_2]\) and \(\phi(x)^T \phi(x') = (x^T x' + 1)^3\), we expand and get

So the corresponding feature vector is

Part 7: re-train using cubic feature mapping

For practical reason, we apply the feature mapping to 10-dimensional PCA representation instead of original 784-dimensional data.

With same settings as base-line case (e.g., temperature parameter = 1), the test error = 0.0865.

3. Basic Neural Network

Part 4: improve learning rate

Since currently we have constant learning rate, which is not best when approaching the minimum, one way we can do is to reduce learning rate gradually by expressing it as a function of iterations.

Part 5: too many hidden units

The danger would be overfitting.

Part 6: training and test error evolution against epochs

For training error, it will continue going down with epochs, while for test error, it will first go down and then go up.

Part 7: optimize the number of epochs

One way to do is use portion of training data (maybe 10%) as inner validation data, for each epoch we evaluate the inner validation error, if it starts going up consecutively for say, 10 epochs, we stop the training process.

4. Deep Neural Networks

Part 1: fully-connected neural networks

(a) after 10 epochs, the test accuracy = 0.9172.

(b) My final model architecture (mnist_nnet_fc_improved.py) has 6 hidden layers (each has 512 neurons, activation is ReLU) and 1 output layer (with 10 neurons, activation is softmax). Optimizer is SGD(lr=0.03, momentum=0.65, decay=0.0001). With this architecture, the test accuracy = 0.9842.

Among all the tweaks I’ve tried, increasing the number of hidden layers (from 1 hidden layer to 6), the number of neurons (from 128 to 512), the learning rate (from 0.001 to 0.03, I also modified momentum and decay) help a lot, boosting the test accuracy from 0.9172 to 0.9842.

Changing hidden layer activation from ReLU to tanh doesn’t make too much difference, but switching to signmoid really harms test accuracy. Too high or too low decay in learning rate also can damage the accuracy; I believe current choice of decay of 0.0001 is the sweet spot.

Part 2: Convolutional neural networks

(a) A simplest convolutional neural network was constructed in the following order:

• A 2D convolutional layer with 32 filters of size 3x3

• A ReLU activation

• A max pooling layer with size 2x2

• A ReLU activation

• A 2D convolutional layer with 64 filters of size 3x3

• A ReLU activation

• A max pooling layer with size 2x2

• A flatten layer

• A fully connected layer with 128 neurons

• A dropout layer with drop probability 0.5

• A fully connected layer with 10 neurons

• A softmax activation

The implementation using keras is list below:

model = Sequential()
model.add(Conv2D(filters=32,kernel_size=(3,3),input_shape=(1, X_train.shape[2], X_train.shape[3])))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(filters=64,kernel_size=(3,3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(units=128))
model.add(Dropout(rate=0.5))
model.add(Dense(units=10))
model.add(Activation("softmax"))

The result is quite encouraging, just one epoch, the test accuracy reaches 0.9810. But it does take longer to finish one epoch (48 secs, while 4 secs for the Part 1 fully-connected neural networks).

To speedup the training time and achieve higher test accuracy, I set up CUDA drivers in a accessible GPU and ran the same ConvNN for 15 epochs. Here’s the training log from GPU:

Using Theano backend.
Using cuDNN version 5110 on context None
Mapped name None to device cuda: Quadro K2200 (0000:03:00.0)
Train on 54000 samples, validate on 6000 samples
Epoch 1/15
54000/54000 [================] - 6s - loss: 0.2062 - acc: 0.9346 - val_loss: 0.0555 - val_acc: 0.9822
Epoch 2/15
54000/54000 [================] - 6s - loss: 0.0771 - acc: 0.9771 - val_loss: 0.0408 - val_acc: 0.9878
Epoch 3/15
54000/54000 [================] - 6s - loss: 0.0577 - acc: 0.9820 - val_loss: 0.0351 - val_acc: 0.9887
......
Epoch 12/15
54000/54000 [================] - 6s - loss: 0.0193 - acc: 0.9934 - val_loss: 0.0346 - val_acc: 0.9912
Epoch 13/15
54000/54000 [================] - 6s - loss: 0.0174 - acc: 0.9944 - val_loss: 0.0317 - val_acc: 0.9912
Epoch 14/15
54000/54000 [================] - 6s - loss: 0.0157 - acc: 0.9945 - val_loss: 0.0345 - val_acc: 0.9912
Epoch 15/15
54000/54000 [================] - 6s - loss: 0.0151 - acc: 0.9948 - val_loss: 0.0348 - val_acc: 0.9905
8832/10000 [===========>....] - ETA: 0sLoss on test set:0.0324026019065 Accuracy on test set: 0.9902

It achieves 99.02% of test accuracy after 15 epochs, and each epoch takes only 6 secs.

5. Classification for overlapping multi-digit MNIST

Pre-step: data representation and model arguments

Training set has 40,000 examples, each example is 42x28 size image, which has two labels indicating two overlapping digits in the image. Each label is a vector of length 10.

Testing set has 4,000 examples, each example is also 42x28 size image, which also has two labels indicating two overlapping digits in the image. Each label is a vector of length 10.

y_train[0] is the fisrt digits in the training images, while y_train[1] is the second.

For model.compile, loss='categorical_crossentropy' means the loss function type is cross entropy function for both outputs, optimizer='sgd' means algorithm is using stochastic gradient descent method to learn, metrics=['accuracy'] means accuracy is used as metric, and loss_weights=[0.5, 0.5] means the two labels are equally treated when calculating loss.

For model.fit, X_train, [y_train[0], y_train[1]] are the training data and their labels, epochs=nb_epoch means total epochs for training, batch_size=batch_size means training is using mini-batch gradient descent approach, and verbose=1 is setting the logging verbosity.

Five explored models

model1: mlp (coded in mlp.py)

The architecture is shown below, with one hidden layer of 64 neurons (relu activation).

After training 30 epochs, the test accuracy for 1st label is 0.9378, for 2nd label is 0.9278. Total training time is 3 secs/epoch, total 90 secs (run on NVIDIA Quadro K2200 GPU, same for all the other four models).

model2: mlp2 (coded in mlp2.py)

The architecture is shown below, with 6 hidden layer of 64 neurons (relu activation).

After training 30 epochs, the test accuracy for 1st label is 0.9503, for 2nd label is 0.9429. Total training time is 4 secs/epoch, total 120 secs.

model3: conv (coded in conv.py)

The architecture is shown below. The first Conv2D has 8 filters, and second has 16 filters. SGD is the optimizer.

After training 15 epochs, the test accuracy for 1st label is 0.9828, for 2nd label is 0.9755. Total training time is 14 secs/epoch, total 210 secs.

model4: conv2 (coded in conv2.py)

The architecture is exactly same as model conv. The difference is adam is the optimizer.

After training 15 epochs, the test accuracy for 1st label is 0.9869, for 2nd label is 0.9822. Total training time is 14 secs/epoch, total 210 secs.

model5: conv3 (coded in conv3.py)

The architecture is exactly same as model conv. The difference is that the first Conv2D has 32 filters, and second has 64 filters

After training 15 epochs, the test accuracy for 1st label is 0.9890, for 2nd label is 0.9869. Total training time is 39 secs/epoch, total 585 secs.

Thoughts on model selection

It seems that convolutional neural networks are more effective in learning images than fully connected neural networks. Model mlp and mlp2 can only achieve test accuracy around 0.95, while conv, conv2 and conv3 can easily achieve beyond 0.98.

For fully connected neural networks, adding more hidden layers seems helpful at least from 1 hidden layer to 6 hidden layers.

Sometime optimizer can make a difference as well, e.g., here using adam optimizer is more effective than SGD to reach minimum.

Today I’m going to focus on how to smoothly install the two important tools for any neural network applications to run on GPUs: CUDA and cuDNN. I noticed there’s serveral installation guides online, e.g., NVIDIA official guide, AWS EC2 guide 1 and AWS EC2 guide 2. They either are outdated, missing some key steps or contain unneccessary settings. More importantly for beginners, we want to have some tests to see each major step is gone though correctly and neccessarily.

For this purpose I decided to create this post, whose goal is to install CUDA and cuDNN on Red Hat Enterprise Linux 7 in a more transparent and reasonable way.

Just to emphasize, my situation was:

• I could easily install theano/tensorflow/keras through anaconda binary platform,

• my application can already successfully run on CPUs,

• I only need to make theano/tensorflow/keras detect there’s GPU available

Test Examples Prep

I prepared two python test scripts: example_1 is from theano official documentation, which is easy and fast to test whether we have connected to GPU. example_2 is from keras example cifar10_cnn, which will be used to final check the speedup brought by GPU. Below is the detail of example_1 script.

from theano import function, config, shared, tensor
import numpy
import time

vlen = 10 _ 30 _ 768 # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], tensor.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, tensor.Elemwise) and
('Gpu' not in type(x.op).__name__)
for x in f.maker.fgraph.toposort()]):
print('Used the cpu')
else:
print('Used the gpu')

Python Anaconda Environment Setup

One command can create a conda environment with: theano. Just run on your terminal:

$ conda create -n theano_test -c conda-forge theano

Status of test examples: able to run on CPUs but not GPUs

At this point, we’ll run example_1 script (no need to run example_2) to make sure example_1 can run on CPUs but not on GPUs.

Run example_1 on CPU mode:

(theano_test) [kehang]$ python example_1.py

Result of example_1 for CPU mode:

[Elemwise{exp,no_inplace}(<TensorType(float64, vector)>)]
Looping 1000 times took 4.095498 seconds
Result is [ 1.23178032 1.61879341 1.52278065 ..., 2.20771815 2.29967753
1.62323285]
Used the cpu

Run example_1 on GPU mode:

(theano_test) [kehang]$ THEANO_FLAGS=device=cuda python example_1.py

Result of example_1 for GPU mode:

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "/home/kehang/miniconda/envs/keras_test/lib/python2.7/site-packages/theano/gpuarray/**init**.py", line 164, in <module>
use(config.device)
File "/home/kehang/miniconda/envs/keras_test/lib/python2.7/site-packages/theano/gpuarray/**init**.py", line 151, in use
init_dev(device)
File "/home/kehang/miniconda/envs/keras_test/lib/python2.7/site-packages/theano/gpuarray/**init**.py", line 60, in init_dev
sched=config.gpuarray.sched)
File "pygpu/gpuarray.pyx", line 614, in pygpu.gpuarray.init (pygpu/gpuarray.c:9415)
File "pygpu/gpuarray.pyx", line 566, in pygpu.gpuarray.pygpu_init (pygpu/gpuarray.c:9106)
File "pygpu/gpuarray.pyx", line 1021, in pygpu.gpuarray.GpuContext.**cinit** (pygpu/gpuarray.c:13468)
GpuArrayException: Error loading library: -1
[Elemwise{exp,no_inplace}(<TensorType(float64, vector)>)]
Looping 1000 times took 4.158732 seconds
Result is [ 1.23178032 1.61879341 1.52278065 ..., 2.20771815 2.29967753
1.62323285]
Used the cpu

Installation of CUDA Toolkit and Driver

This part is well documented by NVIDIA official guide except that some small steps are either too brief (can be not so actionable) or outdated. This post will cover all the commands step by step in an actionable way, for detailed explanations one can always refer to the official guide.

Pre-installation Actions

This step is 100% following NVIDIA official guide: Pre-installation Actions. First run the following commands to verify system requirements are met:

# Verify You Have a CUDA-Capable GPU

$ lspci | grep -i nvidia

# Verify You Have a Supported Version of Linux

$ uname -m && cat /etc/\*release

# Verify the System Has gcc Installed

$ gcc --version

# Install Correct Kernel Headers and Development Packages for Red Hat

$ sudo yum install kernel-devel-$(uname -r) kernel-headers-$(uname -r)

Now you go to NVIDIA CUDA Toolkit Downloads. Below is what I chose for my Red Hat EL7 machine. It’s a fresh installation so I didn’t have to deal with conflicting previous installations. But for people having previous CUDA installation, please refer to Handle Conflicting Installation Methods.

Package Manager Installation

As I mentioned early, I chose to download the rpm(local) installer option, so I need to use package manager installation. This step we’ll following 80% of the NVIDIA official guide: Package Manager Installation and add/modify some steps I regard neccessary but not clear in the official guide.

For users choosing runfile, please refer to Runfile Installation.

# Add third-party repository EPEL to yum repolist

$ wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-9.noarch.rpm
$ rpm -ivh epel-release-7-9.noarch.rpm

# Verify you have EPEL now

$ yum repolist

# Search in EPEL for dkms and libvdpau

# which are dependencies of CUDA

$ yum --enablerepo=epel info dkms
$ yum --enablerepo=epel info libvdpau

# Address custom xorg.conf, if applicable

# I don't have xorg.conf before so it's fine

# if you have, please follow official guide

# Install cuda finally

# takes 10 mins or so

$ sudo rpm -i cuda-repo-rhel7-8-0-local-ga2-8.0.61-1.x86_64.rpm
$ sudo yum clean all
$ sudo yum install cuda

Post-installation Actions

You only need to add /usr/local/cuda-8.0/bin (version number can vary, please check) to your PATH environment variable (either by doing as in below or put into your .bashrc file).

$ export PATH=/usr/local/cuda-8.0/bin:$PATH

To verify GPUs can be accessed, three small steps are needed

• restart the machine

• verify the driver version by running cat /proc/driver/nvidia/version

• run deviceQuery cuda sample binary, in steps as follow

# Copy CUDA samples to your personal directory

# so that you have write permission

$ cuda-install-samples-8.0.sh <your-target-directory>
$ cd <your-target-directory>/NVIDIA_CUDA-8.0_Samples

# Compile samples

$ make

# Running deviceQuery

$ cd bin/x86_64/linux/release
$ ./deviceQuery

Here’s what you’ll get after running deviceQuery:

Status of test examples: able to run on GPUs but cuDNN complaints

Run example_1 on GPU mode:

(theano_test) [kehang]$ THEANO_FLAGS=device=cuda python example_1.py

Result of example_1 for GPU mode:

Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
/usr/bin/ld: cannot find -lcudnn
collect2: error: ld returned 1 exit status

Mapped name None to device cuda: Quadro K2200 (0000:03:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float64, (False,))>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.535818 seconds
Result is [ 1.23178032 1.61879341 1.52278065 ..., 2.20771815 2.29967753
1.62323285]
Used the gpu

Hooray! example_1 is able to access to GPU Quadro K2200 and the wallclock has been reduced by a factor of 8 (from 4.15 sec to 0.53 sec.)

But also it shows that cannot find -lcudnn, which will be our next installation part: cuDNN installation.

Installation of cuDNN

This part of installation is relatively easy, and we’ll mainly follow AWS EC2 guide 2. But still some steps of it need modified or better explained.

cuDNN Download

To obtain the cuDNN library, one needs

• create an account to join NVIDIA developer program.
• download cuDNN

I chose cuDNN Library v5.1 for Linux not v6.0 is because latest theano can only utilize up to v5.1 (v6.0 has a conflict with theano, but maybe future theano can cooperate)

cuDNN Unpack and Install

Copy the download onto your machine, and unpack it

$ tar -zxf cudnn-8.0-linux-x64-v5.1.tgz

Copy cuDNN header file and library files to appropriate include and lib64 directories.

# change directory to the unpacked folder of cudnn

$ cd cuda

# copy related files to /usr/local/cuda/lib64 or /usr/local/cuda/include

# -av will keep the symbolic links as is during copying

sudo cp -av lib64/_ /usr/local/cuda/lib64/
sudo cp -av include/_ /usr/local/cuda/include/

# Update your environment variables in bash session

# or put them in your .bashrc file

export LD_LIBRARY_PATH=/usr/local/cuda/lib64/:$LD_LIBRARY_PATH
export LIBRARY_PATH=/usr/local/cuda/lib64/:$LIBRARY_PATH

Status of test examples: run on GPUs without any complaints

Run example_1 on GPU mode:

(theano_test) [kehang]$ THEANO_FLAGS=device=cuda python example_1.py

Result of example_1 for GPU mode:

Using cuDNN version 5110 on context None
Mapped name None to device cuda: Quadro K2200 (0000:03:00.0)
[GpuElemwise{exp,no_inplace}(<GpuArrayType<None>(float64, (False,))>), HostFromGpu(gpuarray)(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.536293 seconds
Result is [ 1.23178032 1.61879341 1.52278065 ..., 2.20771815 2.29967753
1.62323285]
Used the gpu

Although example_1 doesn’t show time advantage of using cuDNN but it clearly shows it’s using cuDNN smoothly.

For example_2, which is a convolutional neural network application, it can run on GPUs as well (below right-corner video). After switching to GPUs, the training process is sppeded up by at least 6 times (from 291 sec/epoch to 45 sec/epoch, I guess you can immediately tell from the progress bar)

But anyways, hope you can enjoy the installation guide of CUDA and cuDNN on Red Hat. You can leave your comments on Youtube if you have any questions or suggestions.

Here I present our joint work SRS, srs.mit.edu, which is to save us. By using Natural Language Processing and Machine Learning techniques, SRS is able to instantly help summarize customers’ opinions on various aspects of a particular product. On top of that, it also enables users to compare sentiment scores for two similar products.

Workflow

An SRS user is only required to type in a product ID or product URL from Amazon, the rest of work will be taken care of by SRS internal workflow.

Once SRS front-end gets user query, it invokes Review Scraper to work, which collects reviews and stores into SRS Database. Then Aspect Classifier starts to analyze each sentence in the reviews, classifying which aspects the review is discussing (e.g., I like this camera, it can last whole day without re-charging will be classified as battery). Later sentiment analyzer aggregates review positivity for each aspect and send the summary to SRS front-end to present. A typical summary box plot is shown below.

Review Scraper

Once user requests for a product, Review Scraper will first be triggered if the product is not recorded before in database. By using python-amazon-simple-product-api, Review Scraper is able to scrape reviews page by page.

This process can be fairly long especially for products with thousands of reviews. A time limit of 30 seconds is set and top helpful reviews are first scraped so that users can get most relevant information within reasonable time. In order to make reviews gradually complete, Review Scraper is able to continue scraping from where previous scraping stops. Once a product’s reviews are considered complete (a certain ratio between number of reviews in database and total number of reviews online), future requests for that product don’t trigger Review Scraper any more.

Aspect Classifiers

One review usually contains many points, covering more than one aspects of a product. One of the biggest values this project creates is provide sentiment scores for each aspect of the product so that users are informed in a much deeper level compared with given an overall score. So the most crucial part of SRS is to classify each sentence into several aspects.

Currently, we’ve designed an extensible classification framework with three interchangeable classifiers: maxEntropy, word2vec, word2vec_svm.

maxEntropy

Maximum entropy is a supervised learning algorithm that often used in text classification. The idea of maximum entropy in classification is similar to that of the Naive Bayes that they are both discriminative model that estimate the conditional probability

of each predefined product aspect given a sentence/fragment from review. Unlike Naive Bayes, maximum entropy does not hurt by strong independence assumption in the presence of overlapping features that defines the product aspect. For example, if the word shoot is a feature for product aspect picture and video, then maximum entropy could naturally handle this during training of parameters.

The conditional probability in maximum entropy is a parameterized exponential distribution or more precisely it called maximum entropy probability distribution.

Where \(\lambda_i\) is a vector of parameters for aspect \(i\), \(f_i\) is a vector of feature. The choice of feature is arbitrary and the text classifier will yield the best result when chosing key words relevant to the product aspect. Currently, we defined the feature space based on first 20 key words in the tf-idf ranking and the their values are simply the occurrence of the chosen key words. Choice of feature space can definately exploited to further improve classification in the future. During the training process, the parameter matrix \(\Lambda\) is to be determined by minimizing the negative log of the sum of conditional probability for all sentences assuming all sentences are independent of each other.

The maximum entropy has yields a satisfactory accuracy (>60%) that is used as a benchmark for comparing other text classification algorithms.

Sentiment Analyzer

After classification, all the review sentences are grouped by aspects. Sentiment Analyzer is designed to go through sentences aspect by aspect and assigns sentiment scores for each single sentence. Eventually each aspect has a distribution of sentiment, which makes it ready for final rendering in SRS front-end as well as comparison with another product if necessary.