Search API with Google Custom Search and .NET 6

Posted: 8/5/2022

Search option is an important part of every website. Building an appropriate algorithm on your own and adjusting it to the client’s requirements is very time-consuming and expensive, and the results obtained are not always satisfactory. Fortunately, there are ready-made solutions on the market, such as Programmable Search Engine from Google. In this article, I will show how to build our own search engine and integrate it into .NET 6 minimal API.

Technologies used:

.NET 6 with minimal API
Google.Apis.CustomSearchAPI.v1 1.57.0.2455
Google.Apis.Customsearch.v1 1.49.0.2084

Create a Google Custom search engine

To consume our own search engine we need two things - API key and Custom Search Engine ID. Both of them are really easy to get.

Generate API key

To generate an API key, go to Custom Search JSON API: Introduction page and click Get a Key. You have to authenticate with your Google account and then select a project where the key will be assigned. And that’s all you have to do. Click on the Show key button and save the generated key somewhere - we will need it later.

Popup with generated Google Custom Search API key.

Create Custom Search Engine

The Custom Search Engine ID needs a bit more setup. Go to Programmable Search Engine by Google and click on Get started button. You will again be asked for credentials to your Google account and then redirected to the Programmable Search Engine Control Panel. From this view, you can manage your custom search engines. Click on the Add button and provide the required information: name (whatever) and search type (specific websites or global search). You can also add some extra options like an image search or a safe search filter. In the example below, I set up an engine that will search on GitHub and Stackoverflow.

Add a new Google Custom Search engine view.

After successful creation, go to Edit mode. From this view, you can add advance options to your search engine and e.g. limit results to the selected region or language. Search for the Search Engine ID field and copy its value somewhere.

Build .NET 6 minimal API

Create .NET 6 Web API project

Start with creating a new project in Visual Studio with ASP .NET Core Web API template. Choose .NET 6.0 as a framework and remember to uncheck Use controllers - we want to use the Minimal API approach.

Project setup in Visual Studio 2022.

After the project is created, we have to install the required Google libraries. Right-click on the project in the solution explorer view, go to Manage NuGet packages, search for Google.Apis.Customsearch.v1 and Google.Apis.CustomSearchAPI.v1 and install them.

Installed Google libraries in the NuGet Package Manager.

appsettings.json configuration

As we know from the previous section of the article, in order to properly connect to the Search API, we need API Key and Custom Search Engine ID (called CX). It is a good practice to move the configuration part outside of the code, so we create a dedicated section in the appsettings.json file:

"GoogleCustomSearchApi": {
  "ApiKey": "api_key",
  "Cx": "custom_search_engine_id"
}

To load the configuration during application startup, add the code snippet in the Program.cs file:

builder.Services.Configure<GoogleCustomSearchApiConfiguration>(builder.Configuration.GetSection(GoogleCustomSearchApiConfiguration.Name));

It will automatically scan for the GoogleCustomSearchApi section in the appsettings.json file and map it into the GoogleCustomSearchApiConfiguration:

public class GoogleCustomSearchApiConfiguration
{
    public const string Name = "GoogleCustomSearchApi";

    public string ApiKey { get; set; } = string.Empty;
    public string Cx { get; set; } = string.Empty;
}

Then, using IOptions pattern we can inject the configuration wherever it will be needed.

Service service implementation

Domain model

We will start with creating a domain model. We need only two objects:

GoogleSearchResultItem - represents a single item from the results.
GoogleSearchResult - object that contains a collection of single items and some additional properties, like a total number of results.

These objects will be used only for reading mapped data retrieved from the API. In the project root, create a Domain.cs file with both data models (I prefer storing simple, related models in the same file):

public record GoogleSearchResult(int TotalResults, IEnumerable<GoogleSearchResultItem> Items);
public record GoogleSearchResultItem(string Headline, string Description, string Url);

Interface

We start with creating an interface. It is nothing new - needs to have only one asynchronous method SearchAsync with only one (so far) parameter - search phrase. As the result of the method, we want to receive the GoogleSearchResult domain model which we created in the previous step.

public interface IGoogleCustomSearchApiService
{
    Task<GoogleSearchResult> SearchAsync(string searchPhrase);
}

Service

Now it is time for the implementation. Create a new class GoogleCustomSearchApiService which implements the interface created in the previous step. The next step is to inject the configuration using the IOptions pattern which I mentioned before. The code of the service class looks as above:

internal class GoogleCustomSearchApiService : IGoogleCustomSearchApiService
{
    private readonly GoogleCustomSearchApiConfiguration _configuration;

    public GoogleCustomSearchApiService(IOptions<GoogleCustomSearchApiConfiguration> options)
    {
        _configuration = options.Value;
    }

    public async Task<GoogleSearchResult> SearchAsync(string searchPhrase)
    {
        ...
    }
}

Start with creating a CustomSearchService object (from Google.Apis.Customsearch.v1 library). As a constructor parameter, it uses BaseClientService.Initializer where we have to add our ApiKey taken from the configuration object (lines 3-6).

Next, add the request parameters:

Cx - from configuration object
Q - stands for query - this is our search phrase

and execute the request with the ExecuteAsync method (lines 8-16).

If everything was executed correctly, we can map the response from the Google API to our domain model (line 18) and return the result.

public async Task<GoogleSearchResult> SearchAsync(string searchPhrase)
{
    var searchService = new CustomsearchService(new BaseClientService.Initializer
    {
        ApiKey = _configuration.ApiKey
    });

    var listRequest = searchService.Cse.List();
    listRequest.Cx = _configuration.Cx;
    listRequest.Q = searchPhrase;

    var results = await listRequest.ExecuteAsync();
    if(results == null)
    {
        throw new ArgumentNullException(nameof(results));
    }

    var items = results.Items?.Select(x => new GoogleSearchResultItem(x.Title, x.Snippet, x.Link)) ?? new List<GoogleSearchResultItem>();
    return new GoogleSearchResult(int.Parse(results.SearchInformation.TotalResults), items);
}

Service registration

The last thing to do is to register our interface and its implementation in the dependency injection container. We come back again to the Program.cs file and add the given code snippet:

builder.Services.AddTransient<IGoogleCustomSearchApiService, GoogleCustomSearchApiService>();

Endpoint setup

With minimal API setting up the endpoint is easier than ever. We do not need a controller anymore - we can just define the method type (e.g. GET as above), route (“/search”), and define what the given endpoint should do as a lambda function directly in the Program.cs.

app.MapGet("/search", async (string searchPhrase, IGoogleCustomSearchApiService service) =>
{
    return await service.SearchAsync(searchPhrase);
})
.WithName("GetSearchResults");

As you can see above, thanks to the conventions in Minimal API, the framework will automatically resolve searchPhrase as the name of the query parameter and inject the service we registered before.

And that is all. We can run our API and test it directly in the browser thanks to the Swagger:

Example response from the created API executed in Swagger.

Extending the service

Google Custom Search API offers many parameters to customize our search results. We can e.g. limit them to a specific language (Lr), country of origin (Cr), add SafeSearch filtering, or boost results based on the geolocation of the user (Gl). The full list of available attributes is available here - Google.Apis.CustomSearchAPI.v1 - Class CseResource.ListRequest.

One of the most useful parameters is undoubtedly Num and Start. They control the number of returned results (by default it is 10 and cannot be more - it is a limitation added by Google) and the starting index. Using them, we can easily add pagination to our API.

public async Task<GoogleSearchResult> SearchAsync(string searchPhrase, int pageNumber, int pageSize)
{
    ...
    var listRequest = searchService.Cse.List();
    listRequest.Cx = _configuration.Cx;
    listRequest.Q = searchPhrase;
    listRequest.Num = pageSize < 10 ? pageSize : 10; // Number of results (cannot be more than 10)
    listRequest.Start = (pageNumber - 1) \* pageSize; // Start index
    listRequest.Lr = "lang_en"; // Only EN language results
    ...
}

Github

The full project code is available on my Github profile: https://github.com/kubawajs/Google.CustomSearch.API.

Sources

Hero photo by Markus Winkler on Unsplash