Recently, there was a discussion on the mailing list about how a DOS program can detect if it is running under Windows:

I consider the environment string testing to be 100% reliable. That information came from M$ programs... Has anyone tested it under NT and ME

If some one is willing to test, let me explain the method: Windows creates an environment string "windir=C:\WINDOWS" pointing to the windows directory. Note that the variable name is lower case making it difficult for ordinary programs to change it. This variable is removed when Windows exits.

It is reliable up until Windows ME. The default environment has not changed yet.

Though I haven't used this under NT, here's a routine from a batch file I've used since 1997. Porting all this to C should be non-trivial. This used to be part of a set of batch files that made Windows boot to DOS and back without a boot manager.

If anyone needs the whole batch file, I'll post it up on my site. It even included a simple which command to traverse the PATH looking for a file.

  REM  * Try to detect Windows 95/98 ...
  if "%winbootdir%"=="C:\WINDOWS" if "%windir%"=="C:\WINDOWS" goto winyes
  if not "%winbootdir%"=="C:\WINDOWS" if not "%windir%"=="C:\WINDOWS" goto
  if "%winbootdir%"=="C:\WINDOWS" if not "%windir%"=="C:\WINDOWS" goto winlod
  if not "%winbootdir%"=="C:\WINDOWS" if "%windir%"=="C:\WINDOWS" goto winlod
  echo  Windows variables %%WinBootDir%% and %%WinDir%% detected.
  echo  Windows 95/98 GUI is loaded.
  goto windetend
  echo  Windows variables %%WinBootDir%% and %%WinDir%% not detected.
  echo  Windows 95/98 GUI is NOT loaded.
  goto windetend
  echo  Windows variables %%WinBootDir%% OR %%WinDir%% not detected.
  echo  Windows 95/98 GUI detected but is it loaded???
  goto windetend
To further check the system I also used this routine.
  REM  * The following checks for the device drivers HIMEM.SYS and EMM386.EXE
  REM  * in memory.
  if exist XMSXXXX0 echo  HIMEM.SYS is managing extended memory.
  REM ## This the string for DOS 7.0 & 7.1
  if exist EMMQXXX0 echo  EMM386 for DOS 7.1 is managing upper memory blocks.
  REM ## While this is the string for WIN98
  if exist $MMXXXX0 echo  EMM386 for WIN 98 is managing upper memory blocks.
  REM ## This is the string for IFSHLP.SYS, the file system driver
  if exist IFS$HLP$ echo  IFSHLP.SYS is loaded.
  REM ## This checks for the MS CD Extensions.
  if exist MSCD001  echo  MSCDEX is managing the first CD-ROM.
How does this work?

The answer also explains why windows crashes when you use c:\con\con as an argument in explorer. All device drivers have strings in the environment and these can be tested for existence just like any file under windows. I did not know then that windows device drivers have different device strings than the corresponding dos drivers.

-Paolo Carballo

The variable WinBootDir does not exist in Win 3.11 or before. If the matter is just detecting if it is a DosBox in ANY Windows I have alwways used this: (note that the correct sintax is lower case even though this may important only for SUPPL which is case sensitive)
  if not "%windir%"=="" echo This is a Windows DosBox...
which in C translates to
  if (getenv("windir")){
      printf("This is a Windows DosBox...");


Arkady also posts this code snippet:
; Определяется наличие Windows, DOSSHELL или OS/2

code    segment
        assume cs:code,ds:code,es:code,ss:code
        org 100h

;        AX = 160Ah
;Return: AX = 0000h if call supported
;            BX = version (BH=major, BL=minor)
;            CX = mode (0002h = standard, 0003h = enhanced)

        mov     ax, 160Ah
        int     2Fh
        or      ax, ax
        mov     al, bh          ; return major version number of Win3.1
        jz      return          ; if AX=0 (Win3.1 present)

;        AX = 1600h
;Return: AL = status
;            00h neither Windows 3.x enhanced mode nor Windows/386 2.x running
;            01h Windows/386 2.x running
;            80h XMS version 1 driver installed (neither Windows 3.x enhanced
;                  mode nor Windows/386 2.x running) (obsolete--see note)
;            FFh Windows/386 2.x running
;        AL = anything else
;            AL = Windows major version number >= 3
;            AH = Windows minor version number

        mov     ax, 1600h
        int     2Fh
        test    al, 7fh         ; if not Win3.x, Win/386 or XMS code...
        jnz     return          ; ...return Win major version number

;        AX = 4680h
;Return: AX = result
;            0000h MS Windows 3.0 running in real (/R) or standard (/S) mode,
;                  or DOS 5 DOSSHELL active
;            nonzero  no Windows, Windows prior to 3.0, or Windows3 in enhanced
;                  mode

        mov     ax, 4680h
        int     2Fh
        neg     ax              ; negate value and set Carry if arg not zero
        sbb     ax,ax           ; bool(ax): ax==0 -> 0, ax!=0 -> 0ffffh
        inc     ax              ; ax=1 when Win or DOSSHELL
        jnz     return          ; if(ax == 0) return !ax;

;        AX = 4010h
;Return: AX = 4010h if OS/2 not installed
;        AX = 0000h for OS/2 Warp 3.0
;        BX = OS/2 version if installed
;Note:   OS/2 Warp 3.0

        mov     ax, 4010h
        int     2Fh
        sub     ax, 4010h
        or      al, ah          ; al != 0 when OS/2

        mov     ah, 4ch
        int     21h

db 0dh, 0ah, 'Multitask environment (Windows, DOSSHELL, OS/2) detection.'
db 0dh, 0ah, '(c)1999 Arkady Belousov aka Hemul aka'

code    ends
        end     start
@echo off
if errorlevel 1 echo Windows present!
if not errorlevel 1 echo No multitask!

Ralf Quint comments:

The problem with the INT2F code is that it won't work to detect if you're running on Windows NT 4.0 (at least with latest SP's), Windows 2000 or Whi$tler. The call returns that no Windows is installed. In these cases you can only rely on an addtional check if the "true" DOS version is 5.50, than you know that you are running in a NT DOS/Command prompt box...... So far, i haven't been able to find a reliable method to distinguish between the different "flavours" of the NT type Windows OS yet.....