Need suggestion on situation that make calls shell script and that script calls another make

Question

I have a project which uses makefile as entry but at some lower level it calls complex shell scripts as part of jobs while inside those scripts another level of makefiles are called. Below is a simplified example.

In real case, the make call inside shell script takes long time and we hope its jobs could be distributed by emake. But if we use below pattern, "cat shell.out" may run before the "emake -f shell.mk" completes, which will generate error. Is there any way could make "cat shell.out" run after the "emake -f shell.mk" completes but still keep "emake -f shell.mk" distributed other than convert test.sh into a makefile? The real shell script is pretty complex which is not easy to be converted.

Entry Makefile:

    all:testc testd
    	@echo "Running a subshell below.."
    	./test.sh
    	@echo "Exited from test.sh"
    clean:
    	$(RM) -rf shell.out
    testc:
    	@echo "I am testc and sleeping for 2 secs.."
    	sleep 2
    testd:
    	@echo "I am testd and sleeping for 3 secs..."
    	sleep 3

Contents of test.sh:

    #!/bin/sh
    echo "Entering shell"
    emake -f shell.mk
    echo "submake completed!"
    cat shell.out

And then shell.mk:

    all:testa testb
    	@echo "Writing shell.out"
    	@echo "Hello" > shell.out
    testa:
    	@echo "I am testa!"
    	sleep 3
    testb:
    	sleep 2
    	@echo "I am testb!"

Answer 1

In general this will not work because of the way ElectricMake handles recursive make invocations. This is known as the "submake stub" problem -- emake uses a stub to stand in for the actual recursive make invocation, giving the caller the impression that the submake has completed instantaneously, before the submake has actually been processed. Meanwhile, the recursive make is actually processed in parallel, integrated into the dependency graph created by the initial make invocation.

This is an important component of emake's parallel performance. Without this tweak, this common pattern would be severely impacted:

    all:
            for dir in util client server ; do \
                $(MAKE) -C $$dir ; \
            done

The jobs in each submake could be run in parallel, but you would not get parallelism _across_ submakes. That's how gmake works now, by the way.

When the recursive make is invoked directly by emake, there are some additional smarts that enable emake to handle that situation more intelligently, to avoid submake stub problems in many cases. For example:

    all:
            $(MAKE) -C sub foo
            cp sub/foo ./foo

The `cp` command gets automatically split off into a new job (called a "continuation job"), which can be managed and scheduled independently from the submake, and which emake knows should follow the submake in serial order.

But when you put the submake invocation into an external shell script, all that emake sees at the makefile level is a single command:

    all:
            sh dostuff.sh

So there's no opportunity to do anything clever. Fundamentally the lowest level of granularity in emake is an individual command from a makefile rule -- there is no introspection into the content of those commands.

In short, you have two options:

Move the shell script into the makefile directly, as you suggested already.
Don't use a cluster-based emake for the toplevel make invocation -- depending on the structure of your build, you may get good performance by using cluster-based emake only for the submakes, and not for the toplevel invocation. YMMV, of course.

Need suggestion on situation that make calls shell script and that script calls another make

1 Reply

Your answer

Follow this Question

Related Questions