Model Search and Filtering in Laravel

Hi guys, as I have written this functionality a few times at this point, and I have been asked how to do it a couple of times, too. That is Laravel searching and filtering, I am going to develop a simple composer package which I can use to allow users to search and filter Laravel (Eloquent) Models using PHP traits which can be used on Laravel controllers.

Assumptions

I am going to work on the assumption that you are comfortable with basics.

I shall also assume you’re fairly comfortable with the Illuminate HTTP Request object, a Controller, and Eloquent Model.

Concept

So the first thing we want to tackle, as a concept, is that the best way, in my experience, is to start with a query, and then modify it as required. This allows us to optionally do almost anything, but equally allows us to do nothing, and simply return an unfiltered set if we want to.

This point would be true outside of Eloquent too, even if working with vanilla SQL.

Starting Off

To start with you want something like this within your Controller.

public function index(Request $request)
{
// Of course you can change this if you don't want to select everything
$query = YourModel::select('*');
}

Now we have the $query object which we can work with as and where we want to.

Filtering Records

Excluding deleted/is deleted, as we’ll cover that later on.

Let’s say you now want to filter results on author_id just to keep things simple.

if(!empty($request->author)){
$query->where('author_id', '=', $request->author);
}

Of course take into account everything here, you may need to do sanitisation or validation or anything else in there.

Multiple Filters, Save Lines

If you have many fields on which you wish to do this I usually find something like this is better

$fields = ['author_id', 'category_id'];
foreach($fields as $field){
if(!empty($request->$field)){
$query->where($field, '=', $request->$field);
}
}

That example assumes the exposure of your field names, you could always use an associative array so you could do “author” maps to “author_id” and so on, if you were so inclined.

Fluffy Search

Now you might want to do a fluffy search, taking the assumption of posts with content and an author name, and a category name (stored within the model) you could do something like this

if(!empty($request->search)){
$searchFields = ['title','content','author_name','category_name'];
$query->where(function($query) use($request, $searchFields){
$searchWildcard = '%' . $request->search . '%';
foreach($searchFields as $field){
$query->orWhere($field, 'LIKE', $searchWildcard);
}
})
}

Handling Trash

You probably want something which allows you to see soft deleted models, if you’re using them. I do something like this

if($request->trash == true){
// Retrieve soft deleted models only, you can only do this if you're using soft deletion on this model
$query->onlyTrashed();
}

Ordering and Limiting

Now to handle ordering and pagination – which Eloquent beautifully does for us, all we need to do is let the user modify it if they want to.

// Set the query ordering
$query->order($request->orderBy ?? 'updated_at', $request->order ?? 'DESC');

$perPage = $request->per_page ?? 25;

// It is a good idea to make sure we cap this
if($perPage > $maximumPerPage){
$perPage = $maximumPerPage;
}

// Final part - get the results
$results = $query->paginate($perPage);

// You will now need to do something with your results, and return some kind of Response - this could be a JSON response or adding the results to the data returned to the view, depending on your context

There’s an assumption here that our default order is updated at, last updated first, and that if not specified, we want to return 25 pages.

Of course you can store this information wherever you want.

In Summary, and Further Reading

Rather than creating a really complicated set of rules to handle all the things you may need from your API calls, start with a query, and then optionally modify that query.

Further Reading

An introduction to Checksums

Hi everyone

This is a quick introduction into checksums, and practically how to use them, at the request of someone through the Ask Johno section of the blog.

I know what a checksum is, but I’m not sure how to implement or use one (in PHP) to achieve what I need.

Anonymous Asker

They go on to explain what they are trying to achieve, which is essentially to verify a file hasn’t changed (in this case it’s the HTTP document sent by an API) before doing something.

Firstly, a checksum is a very small data snippet (datum) which represents something larger.

In my experience there are 2 main reasons to use one, these are;

  1. To verify a change in something larger
  2. To verify that a piece of data matches from that which was sent by its originator (checksum on an API payload)

I will very briefly cover both of these with PHP examples.

Note; I am using MD5 for simplicity, but depending on your requirements this likely won’t be the best option for your use case.

Edit: Please see Further Reading at the bottom of this article for more information about hashing

Using a Checksum to Detect Changes

Now I don’t know what we’ve got, but whatever it is; we are going to need a string representation, 2 main ways of getting this:

$stringRepresentation = serialize($myThingToCheck);

or

$stringRepresentation = json_encode($myThingToCheck);

Personally, I would advise PHP’s serialize because that will work with, and instantiate objects. However, the choice is yours depending on your use case, JSON is smaller than a PHP serialization.

Now that we have a string, we need to make it small and easy to check. Something like this will work fine;

$checksum = md5($stringRepresentation);

Now to check for changes, we just need the last checksum that we stored that something happened on.

if($checksum != $oldChecksum){
echo 'Something has changed';
}

So that covers how to use a checksum to detect changes in pieces of data, which is useful if you’re having to poll for changes.

Using a Checksum to Verify Validity

This is something I’ve noticed a couple of times, particularly when working in the financial industry, and around certain payment gateways.

It usually looks something like this; but it does change per API integration so be aware of that and follow their own documentation.

$secret = 'my_secret_api_key';

$payload = [
'foo' => 'Bar',
'another' => 'Thing',
'datetime' => '2018-12-25 00:00:00'
];

$jsonPayload = json_encode($payload);

$checksum = md5($jsonPayload . $secret);

$payload['checksum'] = $checksum;

// Do the rest of your stuff here, including sending the payload etc.

One advantage of this approach is that you never expose the API key in plain text.

If you were to want to verify on the API so that you’re the provider, rather than integration, you would simply do these steps in the opposite order, so it would look something like this

// Assuming you've done everything you need, and now have the $payload array/object back

$secret = 'the_secret_youre_expecting';

$checksumProvided = $payload['checksum'];
unset($payload['checksum'];
$checksum = md5(json_encode($payload) . $secret);

if($checksum != $checksumProvided){
// If the checksums don't match, in theory the secret key provided was incorrect
}

I think that about covers this topic, as a brief introduction to checksums, and how they’re often used in PHP.

Further Reading

PHP: hash() for more information about the best ways to hash data

I have deliberately not gotten into the discussion over hashing algorithms, as it really is out of the scope of this article, and indeed a whole book could be written on the topic alone.

Thanks to u/artemix-org and u/BradChesney79 on Reddit for suggesting this edit.

What is a memory leak? A quick analogy

This is something that came up in conversation, some friends and I were discussing deploying code, that runs in the background, to production environments.

One of the things I raised was what can happen with daemon processes, should you have a very small inefficiency, given enough time to run (usually by the time it gets to production) it can, and will, destroy live servers.

I then realised that, at this point in the conversation, a description of what a memory leak is, had become an appropriate thing to explain.

Anybody who knows me, knows I love an analogy; so this is the analogy I gave, to give a really simple explanation as to what a memory leak is:

Every morning, you go to a fast food drive through.

You order a meal, eat it, and throw the paper bag with some leftovers into the passenger footwell.

At the end of the day, you arrive at home. You pick up the bag of rubbish, and put it into your bin. Without realising it, you drop a single french fry in the car.

In development and testing you run this same process 50 times, dropping a single french fry each time. The fries are not visible, they’re under the passenger seat, or with the momentum of the car have ended up in the back.

When you go live, the process runs more frequently, and instead of a single meal you’re buying 10 at a time.

Very quickly, those single french fries culminate in an unusable car, because you can’t fit in a Honda Civic if it has 1,000,000 festering french fries inside.

Matt “Johno the Coder” Johnson, on a cold winter morning

So there it is, a quick explanation of what a memory leak is, in an easy to understand analogy.

I’ve been asked to do walk throughs on practical implementations on daemons and a few other topics, so I am going to write those up soon.

Practically, what might it look like?

Imagine your daemon script looks something like this…

// Store the jobs that have been processed
$jobsProcessed = [];

// This is a daemon script, it needs to run, forever
while(true){

// This is just for demonstration purposes!
$job = getNewJob();

// Do whatever you need to, to handle the job

// Let's store the job we've processed
$jobsProcessed[] = $job;

}

This all looks fairly innocent right. In testing there are probably, at tops, a few thousand test jobs. In production, when this is running forever, that very small array, can become very big. That’s the bit that could cause a server to topple.

For reference, if you do need to keep this information, store it somewhere, anywhere, else. A log file is usually a good shout (as long as you’re periodically cleaning out your log files), perhaps a database (I’d recommend a MyISAM table for this, as you’re dumping a whole load of plain text data). If you keep this information in a variable in your script, it’ll hold in memory, which is exactly where you don’t want it.

So there it is, a quick and easy analogy, with an (overly) simplified example of what it might look like.