What Is a Neural Network?

Why this article exists — and why it's in Stage 4

If you've been reading this knowledge base in order, you've already learned the algorithms that solve the majority of engineering ML problems: Linear Regression, Random Forest, XGBoost, k-NN. You can train a model, evaluate it honestly, and identify which features drive the predictions.

So why learn about neural networks at all?

Two reasons. First, you'll encounter them everywhere — in conference talks, in LinkedIn posts, in conversations with data scientists, in vendor pitches. When someone says "deep learning" or "AI model," they almost always mean neural networks. You need to understand what they are, at least conceptually, to participate in these conversations and push back when the hype doesn't match the problem.

Second, neural networks genuinely are the right tool for some engineering problems — just not the ones most engineers encounter first. If your data is images (visual inspection), raw signals (vibration waveforms), or physics-constrained problems (PINNs), neural networks offer capabilities that tree-based models simply can't match.

This article gives you the conceptual foundation. Not enough to build a neural network from scratch — but enough to understand what they do, when they help, and when they're a waste of time and money.

The basic idea: layers of simple functions

A neural network is built from layers of simple mathematical units called neurons. Each neuron does something almost trivially simple:

1. Take several inputs

2. Multiply each input by a weight

3. Add them up

4. Pass the sum through a function that decides how "activated" the neuron is

5. Send the result to the next layer

If this sounds familiar, it should. A single neuron is essentially a logistic regression — the classification algorithm we mentioned briefly in the Linear Regression article. It computes a weighted sum of inputs and squashes the output through a function.

The power doesn't come from individual neurons. It comes from chaining many neurons together in layers. The output of one layer becomes the input to the next. Each layer transforms the data a little further, extracting increasingly abstract patterns.

A network with:

• An input layer (one neuron per feature — your process parameters)
• One or more hidden layers (where the pattern-finding happens)
• An output layer (the prediction — a number for regression, probabilities for classification)

is called a multilayer perceptron (MLP) or a feedforward neural network. It's the simplest type of neural network, and the easiest to understand.

How a neural network learns

The training process follows the same principle as every other algorithm in this knowledge base:

1. Start with random weights. The network begins knowing nothing.

2. Make predictions. Feed the training data through the network and see what comes out.

3. Measure the error. Compare predictions to actual values using a cost function — the same concept from the Linear Regression article.

4. Adjust the weights. Use gradient descent to nudge each weight in the direction that reduces the error. The specific technique for neural networks is called backpropagation — it efficiently calculates how much each weight contributed to the error, layer by layer, from output back to input.

5. Repeat. Thousands or millions of times, gradually improving the predictions.

This is the same iterative convergence you've seen before — gradient descent walking downhill on the cost surface. The only difference is scale: instead of a few weights (linear regression) or a few hundred split points (trees), a neural network might have tens of thousands or millions of weights to optimise simultaneously.

That scale is both the strength and the weakness. More weights mean more flexibility to capture complex patterns. But more weights also mean more data needed to learn them reliably, more computation to train, and more risk of overfitting.

What makes neural networks different from tree-based models

Understanding the differences helps you decide when each approach is appropriate.

Tree-based models process features independently. A decision tree asks "is temperature > 200°C?" — it looks at one feature at a time and splits the data. This is powerful for tabular data where each column has a distinct physical meaning.

Neural networks blend features together. Each neuron combines all its inputs into a weighted sum. By the second hidden layer, the network is working with combinations of combinations of the original features. This blending is exactly what makes neural networks powerful for images (where the meaning comes from spatial relationships between pixels, not individual pixel values) and exactly what makes them struggle with tabular data (where the meaning lives in individual columns).

Tree-based models learn piece-wise constant functions. They split the feature space into rectangular regions, each with a flat prediction. Sharp thresholds are natural.

Neural networks learn smooth functions. They naturally produce smooth, continuous predictions. Sharp thresholds are harder to learn — the network has to approximate them with many neurons, which requires more data.

Tree-based models ignore irrelevant features automatically. If a feature isn't useful, no tree will split on it. It effectively disappears.

Neural networks are affected by every feature. Every input connects to every neuron in the first hidden layer. Irrelevant features add noise that the network has to learn to suppress — which costs data and training time.

These aren't just theoretical differences. They're the practical reasons why, on the tabular engineering data most of you work with, tree-based models consistently outperform neural networks — as confirmed by the NeurIPS research we've discussed throughout this knowledge base.

When neural networks are genuinely the right tool

Despite the above, there are engineering problems where neural networks aren't just competitive — they're clearly superior:

Image data (computer vision)

Visual quality inspection on a production line. Detecting cracks in X-ray images. Classifying surface defects from photographs. Measuring dimensional features from camera images.

For image data, convolutional neural networks (CNNs) are the standard approach. They're designed to recognise spatial patterns — edges, textures, shapes — by scanning the image with small filters that detect local features and combine them into increasingly complex patterns. No tree-based model can do this.

The practical barrier: you typically need hundreds to thousands of labelled images for each category. A model trained to detect five types of weld defects might need 500+ labelled examples of each defect type. If you don't have that data, transfer learning (starting from a network that was pre-trained on millions of general images, and fine-tuning it on your specific images) can reduce the requirement to as few as 50–100 examples per class.

Sequential / time-series data (raw signals)

Vibration waveforms from accelerometers. Acoustic emission signals during machining. Electrical current signatures from motors. Any data where the pattern exists in the sequence, not just in summary statistics.

For sequential data, recurrent neural networks (RNNs) and transformers can process the raw signal directly and learn patterns that depend on timing, frequency, and temporal context. Tree-based models can work on time-series data too, but they require you to first extract features (RMS, peak frequency, kurtosis) manually — a process that might miss subtle patterns that a neural network would catch.

Physics-informed modelling (PINNs)

Physics-Informed Neural Networks embed physical laws (partial differential equations) directly into the network's cost function. The network learns to satisfy both the data and the physics simultaneously. This is an active research frontier with applications in heat transfer, fluid dynamics, and structural mechanics.

PINNs are exciting because they can learn from very little data when constrained by known physics — addressing the "neural networks need too much data" limitation for problems where the governing equations are known.

Very large datasets with complex interactions

When you have 100,000+ samples and the relationships between features are deeply non-linear with higher-order interactions, neural networks can sometimes capture patterns that even XGBoost misses. But this is rare in most engineering contexts — few teams have datasets this large from their own processes or simulations.

When neural networks are the wrong tool

When your data is a spreadsheet with under 10,000 rows. This describes the vast majority of engineering ML projects. XGBoost will match or outperform a neural network, train in seconds instead of minutes, and give you feature importance for free.

When you need to explain the prediction. "The model predicts this part is defective" is useful. "The model predicts this part is defective because melt temperature contributed +0.23 and injection pressure contributed +0.18 to the defect probability" is far more useful. Tree-based models with SHAP make this easy. Neural networks make it harder — they can be explained with SHAP and other tools, but the explanations are less intuitive and less stable.

When you don't have GPU infrastructure. Neural networks train significantly faster on GPUs than on CPUs — often 10–50× faster. For a quick experiment, CPU training is fine but slow. For anything serious, you need GPU access (a local GPU, cloud computing, or a platform like Google Colab). Tree-based models train happily on any laptop CPU.

When you need rapid iteration. Training a neural network involves choosing an architecture (how many layers, how many neurons), a learning rate schedule, a batch size, regularisation method, and several other settings. The hyperparameter space is vast and the training cycle is slow. XGBoost with early stopping can be tuned in minutes. A neural network experiment might take hours.

When you need to trust the model for safety-critical decisions. In regulated industries (automotive, aerospace, medical), model interpretability isn't optional. Neural networks can be made interpretable, but tree-based models are interpretable by default. Regulatory acceptance is faster with a model whose decision logic can be printed and reviewed.

The honest comparison

Here's a direct comparison on the same tabular engineering dataset — a scenario you might encounter in practice:

	XGBoost	Neural network (MLP)
Data	2,000 rows, 15 features	Same
Training time	8 seconds	3 minutes
Hardware	Laptop CPU	Laptop CPU (GPU would help)
R² on test set	0.91	0.88
Tuning effort	30 minutes (grid search)	3 hours (architecture + hyperparameters)
Feature importance	Built-in, one line of code	Requires additional tools (SHAP)
Explainability	High — SHAP waterfall plots	Moderate — SHAP works but less intuitive

XGBoost wins on every practical dimension: faster training, better accuracy, less tuning, easier interpretation. The neural network isn't bad — 0.88 is a solid score. But it took more effort to get a worse result.

This comparison isn't cherry-picked. It's representative of what happens on typical engineering tabular datasets, and it's consistent with the benchmarking research.

Change the data to images, raw signals, or datasets with 100,000+ rows, and the comparison reverses. Neural networks aren't better or worse — they're designed for different problems. We explore this decision in depth in the next article.

The vocabulary you need

When you encounter neural networks in conversation, here are the terms worth knowing:

Epoch: one complete pass through the training data. Neural networks typically train for tens to hundreds of epochs. (Trees train in a single pass — much faster.)

Batch size: how many samples the network processes before updating its weights. Smaller batches = noisier but more frequent updates. Larger batches = smoother but slower convergence.

Activation function: the function applied after each neuron's weighted sum. ReLU (Rectified Linear Unit) is the most common — it simply outputs zero for negative inputs and passes positive inputs through unchanged. This simple nonlinearity, repeated across thousands of neurons, is what allows networks to learn complex patterns.

Dropout: a regularisation technique where random neurons are temporarily "switched off" during training. This prevents any single neuron from becoming too important and forces the network to learn redundant representations. It's the neural network equivalent of the feature randomisation in Random Forest.

Architecture: the overall structure of the network — how many layers, how many neurons per layer, what type of layers. Choosing an architecture is more art than science, and it's one of the reasons neural networks require more expertise to use effectively.

Key takeaways

A neural network is layers of simple functions chained together. Each neuron computes a weighted sum and applies an activation function. Chaining thousands of these together allows the network to learn complex patterns.

Neural networks shine on images, signals, and very large datasets. For data that has spatial structure (images), temporal structure (waveforms), or deep non-linear interactions across 100,000+ samples, neural networks offer capabilities tree-based models can't match.

For tabular engineering data, tree-based models are usually better. Faster training, less tuning, better interpretability, and often higher accuracy on spreadsheet-style data with under 10,000 rows. This is where most engineers work.

The barrier to entry is higher. Neural networks need more data, more computation (ideally GPUs), more hyperparameter tuning, and more expertise. These aren't insurmountable barriers — but they are real costs that need to be weighed against the benefits.

You don't need to build neural networks to benefit from understanding them. Knowing what they are, when they're appropriate, and when they're overkill makes you a better ML practitioner — even if you never train one yourself.