= Technical Specification =

Weed Audio Extension 1.1 final version.


(C) G. Finch 2005 - 2019

Version 1.1 - added "is_volume_master" to assist with host automation
of audio

22 Nov 2011 - clarified what kinds of channel patterns expected

05 Apr 2012 - if plugin has no audio it should send silence.



The Weed Audio Extension adds audio capabilities to the standard
WEED_PLANT_CHANNEL_TEMPLATE and WEED_PLANT_CHANNEL.




Generally, the following rules apply:

- an AUDIO channel template is declared in the plugin's weed_setup() function by setting the leaf "is_audio"
  to WEED_TRUE. 

- audio is passed as arrays of (signed) float, with values nominally between 1.0 and -1.0

- each Weed audio channel may contain one or more audio subchannels (e.g mono, stereo, 5.1 etc).
  The default is stereo (left, right) | (mono) (center) 

- if a filter cannot process the audio as supplied, then the input audio data MUST be copied to the output unaltered
  (unless the filter is running INPLACE and src == dst, in which case copying is unnecessary).

  The filter must not: alter the audio rate, apply an overall volume adjustment (unless it is part of its general processing - e.g
      	     	       	     	 to avoid clipping), or change the channel layout,
      	     	       	     	 at the output (i.e they must all be the same values as the input) unless specific flags / leaves are set.

  Everything else which follows is OPTIONAL.



- mixing video and audio within a single channel_template is not permitted. However, filters may contain any combination of video and
  audio channels, provided the video and audiio channels have separate templates. The host should follow the same rules
  for creating channels (ordering and number) regardless of whether a channel contains video or audio.

 The only major difference is the timing of processing cycles:

- the process_func() of purely audio filters is usually called more frequently and at different times than for filters with video channels

  the host should update ALL of the (non disabled) channels for a plugin before running the processing cycle,
  and the plugin in turn should update ALL of the non disabled output channels. This means that if a filter has BOTH video
  and audio channels, then the host will likely run the plugin at the video rate and send and expect to receive comparatively
  larger packets of audio data (generally one video frame's worth) at a time.

- if an audio plugin has out parameters which are updated on each audio processing cycle,
  then due to these rate differences, it is quite likely that more than one audio cycle will occur before
  the video plugin has  a chance to read the values. See below (data passing between audio and video filters) for more
  discussion of this.







---   Channel mapping: ----

For simple plugins, these are not required. However, for more advanced use, the following values are defined
by default in weed-effevts.h:

#define WEED_CHANNEL_LAYOUT_TYPE "default"

#define WEED_CH_FRONT_LEFT 0x00000001
#define WEED_CH_FRONT_RIGHT 0x00000002
#define WEED_CH_FRONT_CENTER 0x00000004

#define WEED_CH_LAYOUT_MONO (WEED_CH_FRONT_CENTER)
#define WEED_CH_LAYOUT_STEREO (WEED_CH_FRONT_LEFT | WEED_CH_FRONT_RIGHT)

#define WEED_CH_LAYOUTS_DEFAULT_MIN1 {WEED_CH_LAYOUT_MONO}
#define WEED_CH_LAYOUTS_DEFAULT_MIN2 {WEED_CH_LAYOUT_STEREO, WEED_CH_LAYOUT_MONO}
#define WEED_CH_LAYOUTS_DEFAULT WEED_CH_LAYOUTS_DEFAULT_MIN2


If required, these values may be extended.

weed/weed-compat.h is an optional include, provided for compatibility with some external libraries.

as of WEED_COMPAT_VERSION 0.12.0, if USE_AVUTIL_CHANNEL_LAYOUTS is defined before including the header,
and libavutils/channel_layouts.h is available, then it will be possible to use libavutil audio channel names and layouts in Weed filters.


For example:
WEED_CH_FRONT_LEFT will be defined as AV_CH_FRONT_LEFT
WEED_CH_LOW_FREQUENCY 			AV_CH_LOW_FREQUENCY



and
WEED_CH_LAYOUT_STEREO will be an alias for AV_CH_LAYOUT_STEREO
WEED_CH_LAYOUT_5POINT1 will be an alias for AV_CH_LAYOUT_5POINT1

etc.

in this case WEED_AVUTIL_CHANNEL_LAYOUTS will be defined, and WEED_CHANNEL_LAYOUT_TYPE will be set to "avutil"

Otherwise,
WEED_CHANNEL_LAYOUT_TYPE will be set to "default" in weed-effects.h









== Filter_class ==

For filters with ONLY audio channels,  the "palette_list" leaf may be omitted or set to NULL.






Optional leaves:

the plugin may define these to inform the host, or it may leave them undefined.

The values in the channel should still be checked, since there is no guarantee that the host will always provide
audio in the format requested.




"max_audio_chans" : WEED_SEED_INT : the maximum number of audio subchannels for each stream
		       		       	   	   	     	   that is contained in a Weed channel.

							. Eg, if set to 1 then each input / output audio channel only works with mono and any additional
							         subchannels will be ignored.


							       The filter may optionally provide an "audio_channel_layout" leaf to override the default
							       assumption. for example:

							       For example:
							       Setting the "max_audio_channels" to 1, AND setting "stereo" as an option   
							       in the "audio_channel_layout" leaf,
							       would imply that Weed channel 0 may be mapped to
							       stereo left, and Weed channel 1 may be mapped to stereo right for a SINGLE audio stream.
							       (as opposed to the usual case where each Weed channel would map to 
							       one ENTIRE (mono in this case) audio stream).
							       Combining this with WEED_FILTER_CHANNEL_LAYOUTS_MAY_VARY
							       permits the plugin to define the (set of) audio subchannel(s) available for each Weed channel
							       and setting some of these to OPTIONAL allows for more flexible setups.



							       If "max_audio_channels" is not set, or the value is <= 0 then max audio channels
							       per stream (Weed channel) is assumed to be 2.

							       If the value is set > 2 then it is MANDATORY to provide the "audio_channel_layout"
							       leaf. If the value is set to 1 then providing an "audio_channel_layout" is optional,
							       as described above.

							       Note: there is no guarantee the host will comply with this. The filter should
							       simply ignore excess audio subchannels is the case that the host passes these.

Note: there is no "min_audio_channels" leaf: either the filter should be designed to handle
							       any number of audio channels up to the max value (implied or specified) 
							       or it should set the (set of) permissible layout(s) in the "audio_channel_layout" leaf.



"channel_layout" :: WEED_SEED_INT64  ::  the plugin may define the default
		       	  		   (set of) audio channel layout(s) for all Weed audio channels for the filter_class.
		       	  		   This is MANDATORY if "max_audio_channels" is set to a value > 2.
					    If absent then WEED_CH_LAYOUT_STEREO, WEED_CH_LAYOUT_MONO is assumed,
					    or if "max_audio_channels" is set to 1, WEED_CH_LAYOUT_MONO.
					    Depending on filter flag settings, this may be overridden for individual channel templates.




"audio_rate" : WEED_SEED_INT : the preferred audio sample rate(s) in Hz. May have multiple values in which case the host may select
	       		       	   	     	   from amongst them. They should be listed in descending order of magnitude.
						   If absent then the host may provide audio data at a sample rate of its choosing.

						   It is recommended NOT to set this leaf unless the filter REQUIRES specific audio rates to operate
						   correctly.


 "min_audio_len"  : WEED_SEED_INT : the plugin may set this for a filter to request that the host pass at least this number of samples per
 		       	   	  audio (sub)channel in the "audio_data", 
 				  This leaf is intended mainly for use by plugins which have no audio output channels (i.e
				  analysers or video filters). Use by filters which convert input audio to output audio is STRONGLY
				  DISCOURAGED.

 "max_audio_len"  : the plugin may set this for a filter to request the host pass no more than this number of samples per
 		       	   	  audio (sub)channel in the "audio_data", i.e any samples above this value will be ignored (dropped) by the
				  filter. This leaf is intended mainly for use by plugins which have no audio output channels (i.e
				  analysers or video filters). Use by filters which convert input audio to output audio is STRONGLY
				  DISCOURAGED.


if "min_audio_len" == "max_audio_len", the the host should attempt to supply the specified number of samples,
   		      	 		     However this may not be possible due to timing / other hardware constraints.








Filter flags


*  WEED_FILTER_AUDIO_RATES_MAY_VARY
   Removes the restriction that all in channels must have equal sample rates.
   Either the host may choose freely for each in channel, or the filter may provide a list of acceptable values in the filter_class
   and or / channel templates.
   If the flag WEED_FILTER_IS_CONVERTER is set, allows resampling between in channels and out channels.


*  WEED_FILTER_CHANNEL_LAYOUTS_MAY_VARY
   Removes the restriction that all in channels must have equal channel layouts.
   Either the host may choose freely for each in channel, or the filter may provide a list of acceptable values in the 
   channel templates.
   If the flag WEED_FILTER_IS_CONVERTER is set, allows remapping of channel layouts between in and out.


 * WEED_FILTER_IS_CONVERTER ::

 has multiple meanings depending on other settings.

 On its own it allows the plugin to remap audio subchannels within the audio layout (e.g swap left / right stereo).
 It also allows the plugin to combine (mix) / split multiple input channels into various output channels (depending on the number
 of in / out channels provided) (i.e channel routing)
 
 However the channel layout and sample rate of each in / out channel must be the same.


- if "is_volume_master" is set for a parameter then the filter is an amplifier / volume control / volume mixer. etc/
  and the filter may also alter the volume.



- if WEED_FILTER_CHANNEL_LAYOUTS_MAY_VARY is set, then the filter may change the layout / number of audio channels 
  between input and output. The "audio_channels" and / or "audio_channel_layout" leaves in each channel_template define the
  set of possible transformations. (e.g mix down stereo to mono).



- if WEED_FILTER_AUDIO_RATES_MAY_VARY is set, then the filter may change (resample) audio between inputs and outputs.

The other flag bits have meanings similar to those for video channels:



* WEED_FILTER_HINT_PROCESS_LAST
   This is a hint to the host that the filter should be run AFTER all other filters have performed their processing.
   Examples would be a video filter that overlays subtitles on the finished frame,
   or an audio filter that mixes the final output of several input audio channels. If there are several filters like this in
   the processing chain, then the host should decide the order to run them in.
   


experimental:
*  WEED_FILTER_HINT_MAY_THREAD
   If an audio  plugin operates in such a manner that its output may be subdivided into consecutive blocks of samples,
   then it may set this hint. Ignored for filters which have no out_channels.

   For filters which have both video and audio output channels, the meaning would be ambiguous, so this flag bit should
   not be used for such filters.
   
   - Similar to the method used for video filters, process_func() will be called several times with the same timecode,
   and an extra "offset" leaf added to the out channel(s). This value gives the value in samples from the start of each
   subchannel in "audio_data" that the thread should work on. "audio_data_length" will contain 2 values, the first is
   the reduced data length for the thread - the thread should only write to output "audio_data" from
   offs to offs + audio_data_lenght - 1. The second value in "audio_data_length" contains the true value.
   As with video threading, the thread with offset 0 is considered the "master thread" and should be the only one which
   updates plugin internal values.


 * WEED_FILTER_NON_REALTIME : as for video


 * WEED_FILTER_HINT_STATELESS : as for video





=== Channel template ===

Additional Mandatory leaves:

"is_audio" : WEED_SEED_BOOLEAN : if set to WEED_TRUE, then this Weed channel is for audio.



=== Optional leaves ===

The values in the channel should still be checked, since there is no guanrantee that the host will always provide
audio in the format requested.

if WEED_FILTER_CHANNEL_LAYOUTS_MAY_VARY was set in the filter_class flags then the following leaves may optionally be
set in any channel template, otherwise the defaults for the filter should be applied.

"audio_channels" : WEED_SEED_INT : the desired number of audio channels for a specific Weed channel.
							       if absent then the defaults for the filter_class will be applied.



"channel_layout" :: 
		       	  				       then this leaf may be used to define the set of
							       permissible audio channel layouts for a specific Weed channel.
							       if absent then the defaults for the filter_class will be applied.
							       If "max_audio_channels" was set to 1, then this leaf may be used
							       define an audio subchannel.
					    


if WEED_FILTER_AUDIO_RATES_MAY_VARY was set in the filter_class flags then the following leaves may optionally be
set in any channel template, otherwise the defaults for the filter should be applied.

"audio_rate" : WEED_SEED_INT : the preferred audio sample rate(s) in Hz.
	       		       	   	     	   May have multiple values and the host may select between them.
	       		       	   	     	   They should be listed in DECREASING order of magnitude.
						   if absent then the host may set the audio rate freely (observing the rules
						   for channel matching).
						  





Channel template flags:


  * WEED_CHANNEL_OPTIONAL
    channel that can be left out at initialization time.
    the host decides not to use the channel, it must set "disabled" to
    WEED_TRUE for the channel. The host must reinit
    the instance if a channel is enabled or disabled after init_func(). (With the usual exception of channels created from
    repeatable channel templates, which may be temporarily disabled).


 *  WEED_CHANNEL_CAN_DO_INPLACE    [[BR]]
    If this flag bit is set, the filter can do inplace operations (in and out channels are equal)
    Hosts can select this mode by setting "audio_data" of this out channel
    equal to "audio_data"  of the corresponding (same number; not counting "disabled" channels) 
    in channel. The values of "audio_data_length", "audio_channels", "audio_interleaf" and "audio_rate" must be equal
    for both channels.
    The flag bit is only valid for OUT channels. 



If the plugin needs reiniting when the audio RATE changes, then it should set the flag bit
WEED_CHANNEL_REINIT_ON_AUDIO_RATE_CHANGE (alias for WEED_CHANNEL_REINIT_ON_SIZE_CHANGE).

If the plugin needs reiniting when the audio CHANNEL LAYOUT changes, then it should set the flag bit
WEED_CHANNEL_REINIT_ON_AUDIO_LAYOUT_CHANGE  (alias for WEED_CHANNEL_REINIT_ON_PALETTE_CHANGE)





== WEED_PLANT_CHANNEL ==



Additional Mandatory leaves for CHANNELS with audio - these values are set by the host:

"audio_data" : WEED_SEED_VOIDPTR : stores a pointer to 32 bit signed float data for
	       			      audio buffer. Values within the buffer are nominally in the range -1.0 <=
	  			           value <= 1.0. [can be NULL if the host has no audio to pass].

"audio_data_len" : WEED_SEED_INT : length of audio_data in samples
                                      [can be 0 if the host has no audio to pass]. The length may vary on each cycle, but if the plugin set
				      audio_data_length in





"audio_channels" : WEED_SEED_INT : number of audio channels in the audio_data : must be >=1 (mono), 2==stereo, etc.


"audio_rate" : WEED_SEED_INT : the audio sample rate in Hz. 


T,B,D
"channel_layout" :: the host may set this to define the channel mapping for the channel.
			       	    	   If absent then it may be assumed 1 == MONO. 2 == STEREO (LEFT, RIGHT)

					    It is envisaged that standard channel layout names will be added to weed-compat.h



Notes:

Audio data should be passed to and from the host as an array of WEED_SEED_VOIDPTR
(cast from float *), with each channel's data in a separate element.

In the case that the host has no audio for a Weed channel, it can pass NULL for the data and 0 for the size.

In the case that the host has no audio for a subchannel (i.e within the same Weed channel)
it should fill the subchannel data element with (float) 0.0. (i.e. silence).


The values for "audio_channels", "audio_rate", and "audio_interleaf" MUST be set equal for all channels, in and out, unless:
if the plugin sets WEED_FILTER_AUDIO_RATES_MAY_VARY in filter flags then
the audio sample rate and hence the audio_data_length for each channel may vary. The plugin can override the filter defaults
by setting individual values in channel templates. Plugins MUST NOT alter the sample rate between input and output UNLESS:

if the filter flag WEED_FILTER_IS_CONVERTER is set then the plugin is a resampling plugin and it MUST convert the sample rate
between each input channel and the corresponding output channel.



if the plugin sets WEED_FILTER_AUDIO_LAYOUTS_MAY_VARY in filter flags then
the audio channel layout for each Weed channel may vary. The plugin can override the filter defaults
by setting individual values in channel templates.
Plugins MUST NOT alter the channel layout between input and output UNLESS:

if the filter flag WEED_FILTER_IS_CONVERTER is set then the plugin  MUST convert the channel layout between in and out.
In addition it may mix multiple audio in channels into the output channels.









== WEED_PLANT_PARAMETER_TEMPLATE ==

"is_volume_master" : WEED_SEED_BOOLEAN :

effects which vary the volume (magnitude) of the audio must set the flag WEED_FILTER_IS_CONVERTER.
They must also provide a volume control for the host.

The plugin is allowed to provide only one such parameter per filter.

Such a parameter must be a float, and must *linearly* adjust the volume produced at the output between 0.0 (silence)and 1.0
(unamplified output)  (i.e it is a simple multiplier of the output.
(the maximum may optionally be set higher to allow for extra amplification).

The plugin may optionally provide additional functionality (e.g pan, eq, filtering, remapping of subchannels left / right, etc).

This leaf can be combined with the flag WEED_PARAMETER_VALUE_PER_CHANNEL
This implies that the parameter has multiple values and that each value maps directly to a Weed input channel.
I.E one value per stream, NOT one value per audio channel within the streams.)

If the filter has repeating channels ("max_repeats"), then the flag WEED_PARAMETER_VARIABLE_SIZE should also be set to
allow the number of values to increase or decrease according to the number of channels created from thet template.

An audio filter may also set the flag WEED_FILTER_HINT_PROCESS_LAST, which would imply that it may be run (for example) as the final
mixing filter before the resulting audio is sent to the audio player, or perhaps to overlay a commentary which ought not to
have other effects applied to it.


"is_transition" :
similar meaning as for video only filters. If the filter provides exactly 2 in audio channels and one out then this parameter
will define the mix between the first in channel and the second provided at the output.

  		    at min the effect is fully "off", reproducing only the audio from channel 0 at the output.
		    at max it is fully "on", reproducing only the audio from the second audio in channel

		    Only one parameter may have this leaf, it must be a single valued parameter of type integer or float,
		    and the plugin must have exactlty two mandatory input channels and mandatory output channel.
		    Otherwise this leaf is ignored.
		    Valid only for in parameters.

		    IF the filter also has video channels the this parameter must also control the video transition.










Passing data between audio and video filters.

    For example:
    - a "beat detector" plugin creates THREE output parameters: a boolean (switch) "beat pulse" parameter,
      a boolean "beat latch" (or an integer "beat count"), and an integer "read count" parameter.

      it also creates an input parameter, "read count" with a default value of zero.

      The beat pulse output may be updated on each audio cycle. This output can be fed into other audio filters by the host.

      The second output parameter is either set to WEED_TRUE when a beat is detected, or the count may be incremented,
      depending on the type of parameter desired.

      The plugin should examine the value of the "read count" input parameter, and if it has increased it can reset the "beat latch"
      and finally, it can copy the new value of "read count" to the output parameter "read count".

      When the video cycle runs, each video plugin can read the value of the second parameter to know if (or how many) beats
      were detected since the last video cycle, It can then take the value from the "read count" output, increment it, and feed it
      back into the audio filter's input "read count", to signal that it has read the value. Obviously if another audio cycle is triggered
      partway through the vidoe cycle, not all video filters will have a chance to read the updated value. If this is an issue then it can be
      set up so that only the first video filter reads and updates the audio filter, and the first video filter mirrors the value read as an
      output parameter for the remaining video filters to read. 

      Another solution might be for a single plugin to create both pure video and pure audio filters. These filters can then be connected
      internally within the plugin.



