Sometimes an actor needs a channel for communication with other systems or humans. This channel might be used to receive commands, to provide info about progress or both. To implement this, we will run a HTTP web server inside the actor that will provide: 

  • API to receive commands 
  • HTML page displaying output data

Running a web server in an actor is easy. Each actor run is available at a unique URL (container URL) which is in the form https://[container-key].runs.apify.net. This URL is available in the actor run object returned by the API and also in the actor run console. 

If you start a web server at a port defined by the environment variable APIFY_CONTAINER_PORT  (default value is 4321), the container URL becomes available and gets displayed in the "Live View" tab in the actor run console:

For more details, see the documentation.

Let's try to build the following actor:

  • The actor will provide an API to receive URLs to be processed
  • For each URL, the actor will create a screenshot
  • The screenshot will be stored in the key-value store
  • The actor will provide a web page displaying thumbnails linked to screenshots and a HTML form to submit new URLs

To achieve this we will use:

We need to create a web server with two paths:

  • /  index path will display a page form to submit a new URL and the thumbnails of processed URLs
  • /add-url will provide an API to add new URLs using a HTTP POST request

First import ExpressJS framework and body-parser. Create ExpressJS app and configure body parsers that will allow us to receive form submissions.

const Apify = require('apify')
const express = require('express')
const bodyParser = require('body-parser');

const app = express()

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

 Now we need to read the following environment variables: 

  • APIFY_CONTAINER_PORT  contains a port number where we must start the server 
  • APIFY_CONTAINER_URL  contains a URL under which we can access the container
  • APIFY_DEFAULT_KEY_VALUE_STORE_ID  is simply the ID of the default key-value store of this actor where we can store screenshots
const { 
    APIFY_CONTAINER_PORT,
    APIFY_CONTAINER_URL,
    APIFY_DEFAULT_KEY_VALUE_STORE_ID,
} = process.env;

Create an array of the processed URLs where the n-th URL has its screenshot stored under the key [n].jpg  in key-value store:

const processedUrls = [];

Define a root path that displays a HTML form allowing us to submit new URLs to the path /add-url defined below:

app.get('/', (req, res) => {
    const pageHtml = `
<html>
    <head><title>Example</title></head>
    <body>
        <form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
            URL: <input type="text" name="url" placeholder="http://example.com" />
            <input type="submit" value="Add" />
        </form>
    </body>
</html>`;  

    res.send(pageHtml);
});

And with a second path that receives the new URL submitted using the HTML form; after the URL is processed, it redirects the user back to the root path.

app.post('/add-url', async (req, res) => {
    const { url } = req.body;
    console.log(`Got new URL: ${url}`);
    processedUrls.push(url);

    // After the URL is added redirect user back to root path
    res.redirect('/');
});

And finally we need to start the web server:

// Start the web server!
app.listen(APIFY_CONTAINER_PORT, () => {
    console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
});

Now if we put all the code together

const Apify = require('apify');
const express = require('express')
const bodyParser = require('body-parser');

const app = express()

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));

const {
    APIFY_CONTAINER_PORT,
    APIFY_CONTAINER_URL,
    APIFY_DEFAULT_KEY_VALUE_STORE_ID,
} = process.env;

const processedUrls = [];

app.get('/', (req, res) => {
    const pageHtml = `

<html>
    <head><title>Example</title></head>
    <body>
        <form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
            URL: <input type="text" name="url" placeholder="http://example.com" />
            <input type="submit" value="Add" />
        </form>
    </body>
</html>`;  

    res.send(pageHtml);
});

app.post('/add-url', async (req, res) => {
    const { url } = req.body;
    console.log(`Got new URL: ${url});
    processedUrls.push(url);
    res.redirect('/');
});

app.listen(APIFY_CONTAINER_PORT, () => {
    console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
});

and run it at Apify, then we can open the "Live View" tab in the actor console with newly created form to submit the URL to your actor. After the URL is successfully submitted, it appears in the actor log:

Now let's update our POST /add-url endpoint to grab a screenshot of the provided URL and store it in the key-value store:

app.post('/add-url', async (req, res) => {
    const { url } = req.body;
    console.log(`Got new URL: ${url}`);

    // Start chrome browser and open new page ...
    const browser = await Apify.launchPuppeteer();
    const page = await browser.newPage();

    // ... go to our URL and grab a screenshot ...
    await page.goto(url);
    const screenshot = await page.screenshot({ type: 'jpeg' });

    // ... close browser ...
    await page.close();
    await browser.close();

    // ... save sreenshot to key-value store and add URL to processedUrls.
    await Apify.setValue(`${processedUrls.length}.jpg`, screenshot, { contentType: 'image/jpeg' });
    processedUrls.push(url);

    res.redirect('/');
});

And update the root path to display the screenshots below the form:

app.get('/', (req, res) => {
    let listItems = '';

    // For each of the processed
    processedUrls.forEach((url, index) => {
        const imageUrl = https://api.apify.com/v2/key-value-stores/${APIFY_DEFAULT_KEY_VALUE_STORE_ID}/records/${index}.jpg;

        listItems += `<li>
    <a href="${imageUrl}" target="_blank">
        <img src="${imageUrl}" width="300px" />
        <br />
        ${url}
    </a>
</li>`;
    });

    const pageHtml = `
<html>
    <head><title>Example</title></head>
    <body>
        <form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
            URL: <input type="text" name="url" placeholder="http://example.com" />
            <input type="submit" value="Add" />
            <hr />
            <ul>${listItems}</ul>
        </form>
    </body>
</html>`;

    res.send(pageHtml);
});

 With that we're done! And our application works like a charm :)



The complete code of this actor is available in the library https://www.apify.com/apify/example-web-server . You can run it there or copy it to your account.

Did this answer your question?